[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:27 dist-test-slave-r8cc krb5kdc[19357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:27 dist-test-slave-r8cc krb5kdc[19357](info): set up 2 sockets
Sep 24 19:26:27 dist-test-slave-r8cc krb5kdc[19357](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 19:26:28.650115 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:28 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:28.760672 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38093
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:28.841270 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:28.841513 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:28.841552 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:28.842710 19370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:28.842774 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:28.842797 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:28.842828 19370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:28.842852 19370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:28.844362 19370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:38093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:28.844645 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:28.844879 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:28.850064 19376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:28.850740 19378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:28.851029 19375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:28.851220 19370 server_base.cc:1034] running on GCE node
I20260924 19:26:28.851444 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:28.851773 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:28.852932 19370 hybrid_clock.cc:648] HybridClock initialized: now 1790277988852916 us; error 44 us; skew 500 ppm
Sep 24 19:26:28 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277988, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:28.855351 19370 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:28.855947 19370 webserver.cc:460] Webserver started at http://127.18.229.190:34195/ using document root <none> and password file <none>
I20260924 19:26:28.856256 19370 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:28.856325 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:28.856498 19370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:28.857662 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "f7033f4b127f4853ae4d1d80f45ddbc9"
format_stamp: "Formatted at 2026-09-24 19:26:28 on dist-test-slave-r8cc"
server_key: "1a42ac317a46614de5b4aa772a7ffe1c"
server_key_iv: "5e3e75562a02e8ea24546a374c49ca32"
server_key_version: "encryptionkey@0"
I20260924 19:26:28.858326 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "f7033f4b127f4853ae4d1d80f45ddbc9"
format_stamp: "Formatted at 2026-09-24 19:26:28 on dist-test-slave-r8cc"
server_key: "1a42ac317a46614de5b4aa772a7ffe1c"
server_key_iv: "5e3e75562a02e8ea24546a374c49ca32"
server_key_version: "encryptionkey@0"
I20260924 19:26:28.859700 19370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:28.860488 19385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:28.860683 19370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:28.860801 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "f7033f4b127f4853ae4d1d80f45ddbc9"
format_stamp: "Formatted at 2026-09-24 19:26:28 on dist-test-slave-r8cc"
server_key: "1a42ac317a46614de5b4aa772a7ffe1c"
server_key_iv: "5e3e75562a02e8ea24546a374c49ca32"
server_key_version: "encryptionkey@0"
I20260924 19:26:28.860909 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:28.870994 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:28.873008 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:28.873165 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:28.877106 19370 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:38093
I20260924 19:26:28.877136 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:38093 every 8 connection(s)
I20260924 19:26:28.877470 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:28.878429 19438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:28.881167 19438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9: Bootstrap starting.
I20260924 19:26:28.881721 19438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:28.881942 19438 log.cc:824] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:28.882572 19438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9: No bootstrap required, opened a new log
I20260924 19:26:28.884178 19438 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38093 } }
I20260924 19:26:28.884367 19438 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:28.884392 19438 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7033f4b127f4853ae4d1d80f45ddbc9, State: Initialized, Role: FOLLOWER
I20260924 19:26:28.884529 19438 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38093 } }
I20260924 19:26:28.884616 19438 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:28.884639 19438 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:28.884711 19438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:28.885010 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 19370
I20260924 19:26:28.885113 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:28.885407 19438 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38093 } }
I20260924 19:26:28.885530 19438 leader_election.cc:302] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7033f4b127f4853ae4d1d80f45ddbc9; no voters:
I20260924 19:26:28.885687 19438 leader_election.cc:288] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:28.885747 19441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:28.885923 19350 external_mini_cluster.cc:1428] Setting key 3068861b506c4b67cf9e805d0055d436
I20260924 19:26:28.885927 19441 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [term 1 LEADER]: Becoming Leader. State: Replica: f7033f4b127f4853ae4d1d80f45ddbc9, State: Running, Role: LEADER
I20260924 19:26:28.885952 19438 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:28.886019 19441 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38093 } }
I20260924 19:26:28.886384 19441 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7033f4b127f4853ae4d1d80f45ddbc9. Latest consensus state: current_term: 1 leader_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38093 } } }
I20260924 19:26:28.886534 19442 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7033f4b127f4853ae4d1d80f45ddbc9" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38093 } } }
I20260924 19:26:28.887280 19441 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:28.887306 19442 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9 [sys.catalog]: This master's current role is: LEADER
W20260924 19:26:28.887913 19458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:28.887982 19458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:28.888064 19459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:28.888265 19459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:28.889694 19459 catalog_manager.cc:1347] Generated new cluster ID: 9e12474bd7d1499aa061dc7d2041752a
I20260924 19:26:28.889743 19459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:28.896922 19459 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:28.897516 19459 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 19:26:28 dist-test-slave-r8cc krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:28.905117 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:28.886958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.18.229.190:38093)
0924 19:26:28.887179 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:28.887188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:28.887700 (+ 512us) server_negotiation.cc:407] Connection header received
0924 19:26:28.890035 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:28.890042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:28.890093 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:28.890221 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:28.892322 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:28.893914 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:28.895547 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:28.896233 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:28.896334 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:28.900921 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:28.900926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:28.900927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:28.900938 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:28.902943 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:28.903506 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:28.903508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:28.903509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:28.903628 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:28.903952 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:28.903953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:28.903953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:28.904264 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:28.904387 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:28.904722 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 19:26:28.904863 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260924 19:26:28.912075 19459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7033f4b127f4853ae4d1d80f45ddbc9: Generated new TSK 0
I20260924 19:26:28.912283 19459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:28.936991 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38093
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:29.018383 19464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:29.018615 19464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:29.018657 19464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:29.019826 19464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:29.019891 19464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:29.019936 19464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:29.021560 19464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:29.021822 19464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:29.022058 19464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:29.027218 19469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:29.027231 19472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:29.027324 19470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:29.027427 19464 server_base.cc:1034] running on GCE node
I20260924 19:26:29.027645 19464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:29.027912 19464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:29.029070 19464 hybrid_clock.cc:648] HybridClock initialized: now 1790277989029051 us; error 31 us; skew 500 ppm
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:29.031562 19464 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:29.032092 19464 webserver.cc:460] Webserver started at http://127.18.229.129:33577/ using document root <none> and password file <none>
I20260924 19:26:29.032366 19464 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:29.032431 19464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:29.032574 19464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:29.033656 19464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "49d84d26c6bb4b838049679dc0a98da7"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "d59ae36ec9e765c45e5bb33ab43fdf37"
server_key_iv: "9525f62fab72465c335a985eb0bad813"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.034029 19464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "49d84d26c6bb4b838049679dc0a98da7"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "d59ae36ec9e765c45e5bb33ab43fdf37"
server_key_iv: "9525f62fab72465c335a985eb0bad813"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.035318 19464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:29.036007 19479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.036188 19464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:29.036284 19464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "49d84d26c6bb4b838049679dc0a98da7"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "d59ae36ec9e765c45e5bb33ab43fdf37"
server_key_iv: "9525f62fab72465c335a985eb0bad813"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.036367 19464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:29.046671 19464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:29.048507 19464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:29.048661 19464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:29.048879 19464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:29.049180 19464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:29.049233 19464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.049278 19464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:29.049305 19464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.055037 19464 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:37173
I20260924 19:26:29.055085 19592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:37173 every 8 connection(s)
I20260924 19:26:29.055390 19464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:29.061674 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 19464
I20260924 19:26:29.061774 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:29.061925 19350 external_mini_cluster.cc:1428] Setting key ffb0c944e3cd4fee747199109e15f51d
I20260924 19:26:29.065426 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.056329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:46957 (local address 127.18.229.190:38093)
0924 19:26:29.056394 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:29.056396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:29.056630 (+ 234us) server_negotiation.cc:407] Connection header received
0924 19:26:29.057177 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:29.057179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:29.057193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:29.057242 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:29.058321 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.058837 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.059825 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.060057 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.060154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.062413 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:29.062414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:29.062415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:29.062421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:29.063928 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.064420 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.064421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.064422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.064509 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.064751 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.064752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.064752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.064870 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:29.064949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:29.065232 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:26:29.065336 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:26:29.065598 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.056329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38093 (local address 127.18.229.129:46957)
0924 19:26:29.056511 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:29.056546 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:29.057063 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:29.057312 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:29.057316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:29.058012 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:29.058203 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:29.058209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.058936 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.058937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:29.059732 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:29.059733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.060156 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.060157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:29.060282 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.060728 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:29.060736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:29.062331 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:29.064029 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:29.064037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:29.064039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:29.064330 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:29.064564 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:29.064565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:29.064565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:29.064681 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:29.064938 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:29.064945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:29.065095 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:26:29.065208 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:26:29.065365 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260924 19:26:29.066105 19593 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38093
I20260924 19:26:29.066216 19593 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:29.066419 19593 heartbeater.cc:511] Master 127.18.229.190:38093 requested a full tablet report, sending...
I20260924 19:26:29.067035 19402 ts_manager.cc:194] Registered new tserver with Master: 49d84d26c6bb4b838049679dc0a98da7 (127.18.229.129:37173)
I20260924 19:26:29.067644 19402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:46957
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:29.093752 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38093
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:29.173134 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:29.173390 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:29.173429 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:29.174610 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:29.174679 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:29.174726 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:29.176234 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:29.176507 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:29.176740 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:29.182050 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:29.182044 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:29.182109 19598 server_base.cc:1034] running on GCE node
W20260924 19:26:29.182039 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:29.182441 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:29.182713 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:29.183876 19598 hybrid_clock.cc:648] HybridClock initialized: now 1790277989183873 us; error 36 us; skew 500 ppm
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:29.186599 19598 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:29.187127 19598 webserver.cc:460] Webserver started at http://127.18.229.130:35413/ using document root <none> and password file <none>
I20260924 19:26:29.187319 19598 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:29.187359 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:29.187448 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:29.188539 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "9cb7c42c1fd8441a9c05244effac8b71"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "1c51874f69749c7101253965932ac36b"
server_key_iv: "9f4a6c914099bdb4a2165ad7b4af30e2"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.188849 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "9cb7c42c1fd8441a9c05244effac8b71"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "1c51874f69749c7101253965932ac36b"
server_key_iv: "9f4a6c914099bdb4a2165ad7b4af30e2"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.190086 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:29.190718 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.190922 19598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:29.191004 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "9cb7c42c1fd8441a9c05244effac8b71"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "1c51874f69749c7101253965932ac36b"
server_key_iv: "9f4a6c914099bdb4a2165ad7b4af30e2"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.191058 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:29.210618 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:29.212512 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:29.212620 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:29.212793 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:29.213063 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:29.213100 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.213124 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:29.213136 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.218564 19598 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41275
I20260924 19:26:29.218592 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41275 every 8 connection(s)
I20260924 19:26:29.218923 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:29.227655 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.219746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:46369 (local address 127.18.229.190:38093)
0924 19:26:29.219811 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:29.219821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:29.220036 (+ 215us) server_negotiation.cc:407] Connection header received
0924 19:26:29.220549 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:29.220550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:29.220560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:29.220607 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:29.221482 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.221975 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.222898 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.223095 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.223152 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.225245 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:29.225246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:29.225246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:29.225252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:29.226515 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.226872 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.226873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.226874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.226925 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.227129 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.227129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.227129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.227227 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:29.227275 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:29.227529 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:26:29.227594 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:26:29.227790 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.219733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38093 (local address 127.18.229.130:46369)
0924 19:26:29.219911 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:29.219947 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:29.220467 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:29.220668 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:29.220671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:29.221228 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:29.221406 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:29.221410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.222051 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.222052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:29.222815 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:29.222817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.223169 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.223170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:29.223266 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.223762 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:29.223770 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:29.225164 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:29.226584 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:29.226589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:29.226591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:29.226791 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:29.226987 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:29.226988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:29.226988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:29.227077 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:29.227278 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:29.227282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:29.227398 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 19:26:29.227497 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:26:29.227607 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260924 19:26:29.228312 19727 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38093
I20260924 19:26:29.228403 19727 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:29.228536 19727 heartbeater.cc:511] Master 127.18.229.190:38093 requested a full tablet report, sending...
I20260924 19:26:29.228791 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 19598
I20260924 19:26:29.228886 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:29.228943 19402 ts_manager.cc:194] Registered new tserver with Master: 9cb7c42c1fd8441a9c05244effac8b71 (127.18.229.130:41275)
I20260924 19:26:29.229075 19350 external_mini_cluster.cc:1428] Setting key 367bad65435eb65b2b0f134fb900e941
I20260924 19:26:29.229596 19402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:46369
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:29.254629 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38093
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:29.341503 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:29.341746 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:29.341785 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:29.342924 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:29.342991 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:29.343036 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:29.344523 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:29.344787 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:29.345006 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:29.350039 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:29.350065 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:29.350109 19732 server_base.cc:1034] running on GCE node
W20260924 19:26:29.350063 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:29.350421 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:29.350657 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:29.351822 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790277989351804 us; error 31 us; skew 500 ppm
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:29.354243 19732 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:29.354734 19732 webserver.cc:460] Webserver started at http://127.18.229.131:40925/ using document root <none> and password file <none>
I20260924 19:26:29.354914 19732 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:29.354954 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:29.355036 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:29.356031 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "575c2392af024cf29e5d215c05974532"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "dbbeb1423a755800ea62661ce9650ab9"
server_key_iv: "473ec01ed6b2ca6e8232d5ba59d84f7f"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.356408 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "575c2392af024cf29e5d215c05974532"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "dbbeb1423a755800ea62661ce9650ab9"
server_key_iv: "473ec01ed6b2ca6e8232d5ba59d84f7f"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.357661 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:29.358297 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.358443 19732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.358539 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "575c2392af024cf29e5d215c05974532"
format_stamp: "Formatted at 2026-09-24 19:26:29 on dist-test-slave-r8cc"
server_key: "dbbeb1423a755800ea62661ce9650ab9"
server_key_iv: "473ec01ed6b2ca6e8232d5ba59d84f7f"
server_key_version: "encryptionkey@0"
I20260924 19:26:29.358592 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:29.374913 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:29.376705 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:29.376811 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:29.376986 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:29.377280 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:29.377313 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.377336 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:29.377383 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:29.383054 19732 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:44537
I20260924 19:26:29.383131 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:44537 every 8 connection(s)
I20260924 19:26:29.383427 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:29.389899 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 19732
I20260924 19:26:29.390009 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:29.390192 19350 external_mini_cluster.cc:1428] Setting key f1949b68105f722ac0484c36c34f2093
I20260924 19:26:29.392452 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.384156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:35215 (local address 127.18.229.190:38093)
0924 19:26:29.384203 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:29.384205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:29.384425 (+ 220us) server_negotiation.cc:407] Connection header received
0924 19:26:29.384871 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:29.384872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:29.384884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:29.384933 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:29.385834 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.386335 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.387394 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.387661 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.387719 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.389716 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:29.389717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:29.389717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:29.389724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:29.391287 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.391668 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.391669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.391670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.391725 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.391897 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.391898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.391898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.391996 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:29.392046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:29.392325 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:26:29.392390 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:26:29.392616 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.384172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38093 (local address 127.18.229.131:35215)
0924 19:26:29.384305 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:29.384339 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:29.384784 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:29.384986 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:29.384989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:29.385542 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:29.385730 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:29.385735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.386426 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.386427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:29.387318 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:29.387320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.387736 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.387737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:29.387833 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.388300 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:29.388307 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:29.389629 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:29.391367 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:29.391373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:29.391375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:29.391604 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:29.391769 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:29.391769 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:29.391770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:29.391850 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:29.392036 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:29.392038 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:29.392197 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 19:26:29.392289 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:26:29.392401 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260924 19:26:29.393112 19861 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38093
I20260924 19:26:29.393209 19861 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:29.393378 19861 heartbeater.cc:511] Master 127.18.229.190:38093 requested a full tablet report, sending...
I20260924 19:26:29.393752 19401 ts_manager.cc:194] Registered new tserver with Master: 575c2392af024cf29e5d215c05974532 (127.18.229.131:44537)
I20260924 19:26:29.394125 19401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:35215
I20260924 19:26:29.402370 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:29.424930 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.417418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.18.229.129:37173)
0924 19:26:29.417606 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:29.417611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:29.417622 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:26:29.417648 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:29.417650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:29.417665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:29.417734 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:29.418311 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.419038 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.419723 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.420031 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.420083 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.422515 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:29.422523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:29.422523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:29.422529 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:29.423913 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.424302 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.424305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.424305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.424358 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.424513 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.424513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.424514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.424629 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:29.424742 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:29.424794 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 19:26:29.424855 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
W20260924 19:26:29.425321 19507 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38678
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:29.447350 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:29.440672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.18.229.129:37173)
0924 19:26:29.440746 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:29.440749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:29.440842 (+ 93us) server_negotiation.cc:407] Connection header received
0924 19:26:29.440938 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:29.440939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:29.440954 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:29.441029 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:29.441586 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.442079 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.442805 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:29.443017 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:29.443102 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:29.445293 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:29.445294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:29.445294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:29.445301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:29.446488 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.446780 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.446781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.446781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.446830 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:29.446992 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:29.446992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:29.446992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:29.447075 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:29.447164 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:29.447231 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:26:29.447287 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 19:26:29.447897 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 19464
I20260924 19:26:29.452597 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 19598
I20260924 19:26:29.457726 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 19732
I20260924 19:26:29.461994 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 19370
2026-09-24T19:26:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1862 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19875](info): set up 2 sockets
Sep 24 19:26:29 dist-test-slave-r8cc krb5kdc[19875](info): commencing operation
krb5kdc: starting...
W20260924 19:26:30.491600 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:30 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:30.602147 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38567
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:30.682003 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:30.682256 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:30.682297 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:30.683434 19888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:30.683511 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:30.683537 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:30.683564 19888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:30.683595 19888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:30.685088 19888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:30.685364 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:30.685602 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:30.690564 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:30.690634 19888 server_base.cc:1034] running on GCE node
W20260924 19:26:30.690558 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:30.690558 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:30.691030 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:30.691322 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:30.692469 19888 hybrid_clock.cc:648] HybridClock initialized: now 1790277990692455 us; error 29 us; skew 500 ppm
Sep 24 19:26:30 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:30.694849 19888 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:30.695370 19888 webserver.cc:460] Webserver started at http://127.18.229.190:33571/ using document root <none> and password file <none>
I20260924 19:26:30.695600 19888 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:30.695645 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:30.695808 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:30.696970 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "fddf9df4c0804cba8f5a529a790b4be5"
format_stamp: "Formatted at 2026-09-24 19:26:30 on dist-test-slave-r8cc"
server_key: "9c9461a45e598252efd4cacf7afab39d"
server_key_iv: "3554a04db8c0f646e510cf8b62ea0733"
server_key_version: "encryptionkey@0"
I20260924 19:26:30.697328 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "fddf9df4c0804cba8f5a529a790b4be5"
format_stamp: "Formatted at 2026-09-24 19:26:30 on dist-test-slave-r8cc"
server_key: "9c9461a45e598252efd4cacf7afab39d"
server_key_iv: "3554a04db8c0f646e510cf8b62ea0733"
server_key_version: "encryptionkey@0"
I20260924 19:26:30.698668 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:30.699352 19903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:30.699517 19888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:30.699615 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "fddf9df4c0804cba8f5a529a790b4be5"
format_stamp: "Formatted at 2026-09-24 19:26:30 on dist-test-slave-r8cc"
server_key: "9c9461a45e598252efd4cacf7afab39d"
server_key_iv: "3554a04db8c0f646e510cf8b62ea0733"
server_key_version: "encryptionkey@0"
I20260924 19:26:30.699697 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:30.722491 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:30.724375 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:30.724555 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:30.728585 19888 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:38567
I20260924 19:26:30.728621 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:38567 every 8 connection(s)
I20260924 19:26:30.728992 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:30.729815 19956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:30.732004 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5: Bootstrap starting.
I20260924 19:26:30.732620 19956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:30.732875 19956 log.cc:824] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:30.733503 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5: No bootstrap required, opened a new log
I20260924 19:26:30.735004 19956 raft_consensus.cc:348] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddf9df4c0804cba8f5a529a790b4be5" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38567 } }
I20260924 19:26:30.735136 19956 raft_consensus.cc:374] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:30.735158 19956 raft_consensus.cc:729] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddf9df4c0804cba8f5a529a790b4be5, State: Initialized, Role: FOLLOWER
I20260924 19:26:30.735299 19956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddf9df4c0804cba8f5a529a790b4be5" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38567 } }
I20260924 19:26:30.735380 19956 raft_consensus.cc:388] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:30.735409 19956 raft_consensus.cc:482] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:30.735440 19956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:30.735914 19956 raft_consensus.cc:504] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddf9df4c0804cba8f5a529a790b4be5" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38567 } }
I20260924 19:26:30.736002 19956 leader_election.cc:302] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fddf9df4c0804cba8f5a529a790b4be5; no voters:
I20260924 19:26:30.736143 19956 leader_election.cc:288] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:30.736236 19959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:30.736408 19956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:30.736436 19959 raft_consensus.cc:686] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [term 1 LEADER]: Becoming Leader. State: Replica: fddf9df4c0804cba8f5a529a790b4be5, State: Running, Role: LEADER
I20260924 19:26:30.736560 19959 consensus_queue.cc:237] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddf9df4c0804cba8f5a529a790b4be5" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38567 } }
I20260924 19:26:30.736971 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 19888
I20260924 19:26:30.737087 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:30.737155 19961 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fddf9df4c0804cba8f5a529a790b4be5. Latest consensus state: current_term: 1 leader_uuid: "fddf9df4c0804cba8f5a529a790b4be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddf9df4c0804cba8f5a529a790b4be5" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38567 } } }
I20260924 19:26:30.737233 19961 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:30.737262 19350 external_mini_cluster.cc:1428] Setting key b6be4b8e7473a878c5fee0e550d099b7
I20260924 19:26:30.737403 19960 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fddf9df4c0804cba8f5a529a790b4be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddf9df4c0804cba8f5a529a790b4be5" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38567 } } }
I20260924 19:26:30.737476 19960 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5 [sys.catalog]: This master's current role is: LEADER
W20260924 19:26:30.738438 19975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:30.738505 19975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:30.738602 19968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:30.738793 19968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:30.740587 19968 catalog_manager.cc:1347] Generated new cluster ID: edbf7e8e055b47c08bc8781dd4a3c3d5
I20260924 19:26:30.740645 19968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:30 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:30.747876 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:30.738064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.18.229.190:38567)
0924 19:26:30.738275 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:30.738285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:30.738332 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:26:30.738464 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:30.738473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:30.738533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:30.739329 (+ 796us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:30.739463 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:30.740490 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:30.741295 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:30.741753 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:30.741883 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:30.744192 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:30.744196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:30.744196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:30.744208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:30.746361 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:30.746740 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:30.746743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:30.746744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:30.746848 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:30.747067 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:30.747068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:30.747068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:30.747295 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:30.747455 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:30.747574 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:26:30.747713 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260924 19:26:30.756862 19968 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:30.757385 19968 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:30.760290 19968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fddf9df4c0804cba8f5a529a790b4be5: Generated new TSK 0
I20260924 19:26:30.760473 19968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:30.795274 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38567
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:30.898794 19982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:30.899011 19982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:30.899039 19982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:30.900394 19982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:30.900463 19982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:30.900513 19982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:30.902215 19982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:30.902503 19982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:30.902770 19982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:30.908304 19988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:30.908281 19990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:30.908281 19987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:30.908514 19982 server_base.cc:1034] running on GCE node
I20260924 19:26:30.908959 19982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:30.909235 19982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:30.910424 19982 hybrid_clock.cc:648] HybridClock initialized: now 1790277990910430 us; error 43 us; skew 500 ppm
Sep 24 19:26:30 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:30.913035 19982 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:30.913479 19982 webserver.cc:460] Webserver started at http://127.18.229.129:39733/ using document root <none> and password file <none>
I20260924 19:26:30.913720 19982 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:30.913786 19982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:30.913930 19982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:30.914970 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "8f5560f12fec45778209e971b30020f5"
format_stamp: "Formatted at 2026-09-24 19:26:30 on dist-test-slave-r8cc"
server_key: "642064cf6a7235540f8bb5bec50d5631"
server_key_iv: "54ff23fc9dbd6d2fc4f8f2ae7f916f57"
server_key_version: "encryptionkey@0"
I20260924 19:26:30.915328 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "8f5560f12fec45778209e971b30020f5"
format_stamp: "Formatted at 2026-09-24 19:26:30 on dist-test-slave-r8cc"
server_key: "642064cf6a7235540f8bb5bec50d5631"
server_key_iv: "54ff23fc9dbd6d2fc4f8f2ae7f916f57"
server_key_version: "encryptionkey@0"
I20260924 19:26:30.916710 19982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:30.917414 19997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:30.917577 19982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:30.917680 19982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "8f5560f12fec45778209e971b30020f5"
format_stamp: "Formatted at 2026-09-24 19:26:30 on dist-test-slave-r8cc"
server_key: "642064cf6a7235540f8bb5bec50d5631"
server_key_iv: "54ff23fc9dbd6d2fc4f8f2ae7f916f57"
server_key_version: "encryptionkey@0"
I20260924 19:26:30.917753 19982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:30.930480 19982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:30.932915 19982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:30.933084 19982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:30.933320 19982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:30.933636 19982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:30.933692 19982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:30.933735 19982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:30.933763 19982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:30.939745 19982 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:45881
I20260924 19:26:30.939826 20110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:45881 every 8 connection(s)
I20260924 19:26:30.940218 19982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:30.941107 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 19982
I20260924 19:26:30.941200 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:30.941349 19350 external_mini_cluster.cc:1428] Setting key 4e0a4ee540581f7e25a19f94ef277c1b
Sep 24 19:26:30 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277990, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
I20260924 19:26:30.950484 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:30.941010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:34547 (local address 127.18.229.190:38567)
0924 19:26:30.941059 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:30.941061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:30.941296 (+ 235us) server_negotiation.cc:407] Connection header received
0924 19:26:30.942516 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:30.942517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:30.942530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:30.942578 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:30.943352 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:30.943852 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:30.944948 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:30.945136 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:30.945193 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:30.947596 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:30.947598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:30.947599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:30.947606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:30.949073 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:30.949512 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:30.949513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:30.949514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:30.949570 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:30.949819 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:30.949820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:30.949821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:30.949943 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:30.950016 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:30.950302 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 19:26:30.950395 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:30.950671 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:30.940993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38567 (local address 127.18.229.129:34547)
0924 19:26:30.941168 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:30.941205 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:30.942382 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:30.942662 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:30.942665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:30.942916 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:30.943239 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:30.943246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:30.943955 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:30.943956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:30.944844 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:30.944847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:30.945216 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:30.945217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:30.945364 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:30.945878 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:30.945889 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:30.947498 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:30.949176 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:30.949183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:30.949185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:30.949423 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:30.949636 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:30.949637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:30.949637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:30.949751 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:30.950018 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:30.950025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:30.950166 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 19:26:30.950269 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:26:30.950428 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260924 19:26:30.951200 20111 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38567
I20260924 19:26:30.951314 20111 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:30.951519 20111 heartbeater.cc:511] Master 127.18.229.190:38567 requested a full tablet report, sending...
I20260924 19:26:30.952044 19920 ts_manager.cc:194] Registered new tserver with Master: 8f5560f12fec45778209e971b30020f5 (127.18.229.129:45881)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
I20260924 19:26:30.952888 19920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:34547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:30.966624 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38567
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:31.050240 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:31.050446 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:31.050470 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:31.051695 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:31.051738 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:31.051793 20116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:31.053359 20116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:31.053613 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:31.053854 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:31.058745 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:31.058805 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:31.058787 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:31.058966 20116 server_base.cc:1034] running on GCE node
I20260924 19:26:31.059197 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:31.059490 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:31.060652 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790277991060639 us; error 46 us; skew 500 ppm
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:31.063077 20116 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:31.063597 20116 webserver.cc:460] Webserver started at http://127.18.229.130:42581/ using document root <none> and password file <none>
I20260924 19:26:31.063846 20116 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:31.063915 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:31.064080 20116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:31.065127 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "837e64f2cb1b4ebab19ab4fb2e652664"
format_stamp: "Formatted at 2026-09-24 19:26:31 on dist-test-slave-r8cc"
server_key: "39fd733ca7df4d74d2fc2fa9abd790c3"
server_key_iv: "6683f0962430740174f0f1668866d391"
server_key_version: "encryptionkey@0"
I20260924 19:26:31.065485 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "837e64f2cb1b4ebab19ab4fb2e652664"
format_stamp: "Formatted at 2026-09-24 19:26:31 on dist-test-slave-r8cc"
server_key: "39fd733ca7df4d74d2fc2fa9abd790c3"
server_key_iv: "6683f0962430740174f0f1668866d391"
server_key_version: "encryptionkey@0"
I20260924 19:26:31.066749 20116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:26:31.067471 20131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:31.067648 20116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:31.067749 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "837e64f2cb1b4ebab19ab4fb2e652664"
format_stamp: "Formatted at 2026-09-24 19:26:31 on dist-test-slave-r8cc"
server_key: "39fd733ca7df4d74d2fc2fa9abd790c3"
server_key_iv: "6683f0962430740174f0f1668866d391"
server_key_version: "encryptionkey@0"
I20260924 19:26:31.067835 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:31.077689 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:31.079414 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:31.079572 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:31.079794 20116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:31.080261 20116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:31.080366 20116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:31.080420 20116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:31.080446 20116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:31.086248 20116 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:44007
I20260924 19:26:31.086299 20244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:44007 every 8 connection(s)
I20260924 19:26:31.086611 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:31.091383 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 20116
I20260924 19:26:31.091518 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:31.091683 19350 external_mini_cluster.cc:1428] Setting key 13d759168df5675ef8d6058381fdbae9
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:31.096724 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.087371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:39517 (local address 127.18.229.190:38567)
0924 19:26:31.087440 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.087441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.087658 (+ 217us) server_negotiation.cc:407] Connection header received
0924 19:26:31.088616 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.088617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.088629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.088672 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:31.089297 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.089797 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.090805 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.091033 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.091119 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.093751 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:31.093753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:31.093753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:31.093770 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:31.095216 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.095645 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.095646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.095647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.095708 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.096010 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.096011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.096012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.096179 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:31.096264 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.096541 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.096628 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:26:31.096939 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.087372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38567 (local address 127.18.229.130:39517)
0924 19:26:31.087536 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.087569 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.088502 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.088740 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.088746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.089007 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:31.089191 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.089194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.089891 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.089893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.090700 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.090702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.091128 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.091129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.091264 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.091818 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:31.091826 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:31.093643 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:31.095316 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:31.095323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:31.095326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:31.095554 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:31.095774 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:31.095775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:31.095775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:31.095927 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:31.096253 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:31.096260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:31.096404 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.096522 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.096688 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260924 19:26:31.097612 20245 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38567
I20260924 19:26:31.097723 20245 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:31.097904 20245 heartbeater.cc:511] Master 127.18.229.190:38567 requested a full tablet report, sending...
I20260924 19:26:31.098328 19920 ts_manager.cc:194] Registered new tserver with Master: 837e64f2cb1b4ebab19ab4fb2e652664 (127.18.229.130:44007)
I20260924 19:26:31.098867 19920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:39517
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:31.116470 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38567
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:31.199517 20250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:31.199725 20250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:31.199748 20250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:31.201004 20250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:31.201046 20250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:31.201073 20250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:31.202569 20250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38567
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:31.202759 20250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:31.202944 20250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:31.207902 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:31.207916 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:31.207916 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:31.208169 20250 server_base.cc:1034] running on GCE node
I20260924 19:26:31.208477 20250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:31.208742 20250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:31.209918 20250 hybrid_clock.cc:648] HybridClock initialized: now 1790277991209907 us; error 36 us; skew 500 ppm
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:31.212610 20250 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:31.213078 20250 webserver.cc:460] Webserver started at http://127.18.229.131:33863/ using document root <none> and password file <none>
I20260924 19:26:31.213326 20250 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:31.213397 20250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:31.213549 20250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:31.214615 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "60ad561caa3847fe8aec9a16dfd573bc"
format_stamp: "Formatted at 2026-09-24 19:26:31 on dist-test-slave-r8cc"
server_key: "c2dac8722e01dd84a6f0107bd6851519"
server_key_iv: "1ea74bfce08bcba69a2ed60c3c7dc7f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:31.214972 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "60ad561caa3847fe8aec9a16dfd573bc"
format_stamp: "Formatted at 2026-09-24 19:26:31 on dist-test-slave-r8cc"
server_key: "c2dac8722e01dd84a6f0107bd6851519"
server_key_iv: "1ea74bfce08bcba69a2ed60c3c7dc7f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:31.216348 20250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:26:31.217044 20265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:31.217221 20250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:31.217330 20250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "60ad561caa3847fe8aec9a16dfd573bc"
format_stamp: "Formatted at 2026-09-24 19:26:31 on dist-test-slave-r8cc"
server_key: "c2dac8722e01dd84a6f0107bd6851519"
server_key_iv: "1ea74bfce08bcba69a2ed60c3c7dc7f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:31.217403 20250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:31.228217 20250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:31.230571 20250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:31.230738 20250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:31.230974 20250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:31.231349 20250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:31.231386 20250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:31.231410 20250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:31.231422 20250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:31.237496 20250 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:42133
I20260924 19:26:31.237540 20378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:42133 every 8 connection(s)
I20260924 19:26:31.237946 20250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:31.241271 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 20250
I20260924 19:26:31.241389 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:31.241554 19350 external_mini_cluster.cc:1428] Setting key e8f0e258042bf7ae8cda3a51fcaf3f33
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:31.247808 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.238661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:58261 (local address 127.18.229.190:38567)
0924 19:26:31.238708 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.238711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.238897 (+ 186us) server_negotiation.cc:407] Connection header received
0924 19:26:31.239871 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.239873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.239887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.239945 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:31.240622 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.241178 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.242621 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.242841 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.242929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.245089 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:31.245090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:31.245091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:31.245098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:31.246562 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.246931 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.246932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.246932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.246981 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.247224 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.247225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.247226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.247350 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:31.247417 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.247657 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.247723 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:26:31.248008 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.238620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38567 (local address 127.18.229.131:58261)
0924 19:26:31.238758 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.238808 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.239753 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.240015 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.240020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.240327 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:31.240519 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.240524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.241281 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.241282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.242535 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.242537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.242938 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.242939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.243073 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.243627 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:31.243643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:31.245003 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:31.246642 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:31.246648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:31.246650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:31.246864 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:31.247039 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:31.247039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:31.247039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:31.247136 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:31.247415 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:31.247419 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:31.247535 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.247617 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.247732 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260924 19:26:31.248529 20379 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38567
I20260924 19:26:31.248652 20379 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:31.248858 20379 heartbeater.cc:511] Master 127.18.229.190:38567 requested a full tablet report, sending...
I20260924 19:26:31.249230 19920 ts_manager.cc:194] Registered new tserver with Master: 60ad561caa3847fe8aec9a16dfd573bc (127.18.229.131:42133)
I20260924 19:26:31.249595 19920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:58261
I20260924 19:26:31.253298 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:26:31.261487 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.255038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.18.229.190:38567)
0924 19:26:31.255096 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.255098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.255195 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:26:31.255270 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.255271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.255281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.255348 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:31.255909 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.256463 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.257193 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.257380 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.257455 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.258671 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:31.258672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:31.258672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:31.258679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:31.260274 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.260683 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.260684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.260684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.260745 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.260945 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.260946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.260946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.261099 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:31.261210 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.261311 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.261390 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:26:31.263846 19920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:31.265537 19920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:31.272534 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.268092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:42133 (local address 127.0.0.1:52254)
0924 19:26:31.268592 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.268602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.268687 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.269042 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.269043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.269045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:31.269103 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.269105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.270061 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.270062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.271258 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.271261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.272225 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.272226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.272310 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.272327 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.272398 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.272458 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260924 19:26:31.272560 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.268204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.18.229.131:42133)
0924 19:26:31.268385 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.268390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.268697 (+ 307us) server_negotiation.cc:407] Connection header received
0924 19:26:31.268863 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.268866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.268881 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.268955 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:31.269196 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.269969 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.271345 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.272156 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.272229 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.272287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.272424 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.272472 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260924 19:26:31.272964 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.268392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.18.229.130:44007)
0924 19:26:31.268547 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.268551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.268669 (+ 118us) server_negotiation.cc:407] Connection header received
0924 19:26:31.268779 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.268782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.268799 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.268877 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:31.269138 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.270257 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.271715 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.272519 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.272589 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.272637 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.272851 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.272891 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260924 19:26:31.273247 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.268092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:44007 (local address 127.0.0.1:34062)
0924 19:26:31.268557 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.268594 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.268684 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.268973 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.268976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.268979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:31.269054 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.269056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.270745 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.270746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.271634 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.271636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.272671 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.272672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.272734 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.272737 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.272832 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.272873 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":103,"threads_started":1}
I20260924 19:26:31.273554 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.268104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45881 (local address 127.0.0.1:40148)
0924 19:26:31.268799 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.268807 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.268852 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.269054 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.269055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.269057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:31.269103 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.269105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.270367 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.270368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.272261 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.272262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.273306 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.273307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.273377 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.273381 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.273436 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.273480 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260924 19:26:31.273727 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.268365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.18.229.129:45881)
0924 19:26:31.268528 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.268533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.268914 (+ 381us) server_negotiation.cc:407] Connection header received
0924 19:26:31.268968 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.268970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.268984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.269059 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:31.269185 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.270269 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.272350 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.273156 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.273219 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.273273 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.273607 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.273659 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260924 19:26:31.273706 20313 tablet_service.cc:1467] Processing CreateTablet for tablet 2af023f4cd6d4a02b24d89d489d8a77b (DEFAULT_TABLE table=test-table [id=0cfd847cc831412a8c88863bf2cd1776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:31.274075 20313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af023f4cd6d4a02b24d89d489d8a77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:31.273968 20179 tablet_service.cc:1467] Processing CreateTablet for tablet 2af023f4cd6d4a02b24d89d489d8a77b (DEFAULT_TABLE table=test-table [id=0cfd847cc831412a8c88863bf2cd1776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:31.274271 20179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af023f4cd6d4a02b24d89d489d8a77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:31.274530 20045 tablet_service.cc:1467] Processing CreateTablet for tablet 2af023f4cd6d4a02b24d89d489d8a77b (DEFAULT_TABLE table=test-table [id=0cfd847cc831412a8c88863bf2cd1776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:31.274869 20045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af023f4cd6d4a02b24d89d489d8a77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:31.277510 20396 tablet_bootstrap.cc:492] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: Bootstrap starting.
I20260924 19:26:31.277892 20397 tablet_bootstrap.cc:492] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: Bootstrap starting.
I20260924 19:26:31.278278 20396 tablet_bootstrap.cc:654] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:31.278509 20397 tablet_bootstrap.cc:654] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:31.278877 20396 log.cc:824] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:31.279120 20397 log.cc:824] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:31.279454 20398 tablet_bootstrap.cc:492] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc: Bootstrap starting.
I20260924 19:26:31.279690 20396 tablet_bootstrap.cc:492] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: No bootstrap required, opened a new log
I20260924 19:26:31.279762 20396 ts_tablet_manager.cc:1397] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:31.280056 20398 tablet_bootstrap.cc:654] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:31.280563 20397 tablet_bootstrap.cc:492] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: No bootstrap required, opened a new log
I20260924 19:26:31.280637 20397 ts_tablet_manager.cc:1397] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:26:31.280671 20398 log.cc:824] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:31.281553 20398 tablet_bootstrap.cc:492] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc: No bootstrap required, opened a new log
I20260924 19:26:31.281626 20398 ts_tablet_manager.cc:1397] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:31.281795 20396 raft_consensus.cc:348] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.281958 20396 raft_consensus.cc:374] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:31.282030 20396 raft_consensus.cc:729] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5560f12fec45778209e971b30020f5, State: Initialized, Role: FOLLOWER
I20260924 19:26:31.282164 20396 consensus_queue.cc:260] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.282474 20396 ts_tablet_manager.cc:1428] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:31.282493 20111 heartbeater.cc:503] Master 127.18.229.190:38567 was elected leader, sending a full tablet report...
I20260924 19:26:31.282771 20397 raft_consensus.cc:348] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.283330 20397 raft_consensus.cc:374] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:31.283406 20397 raft_consensus.cc:729] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837e64f2cb1b4ebab19ab4fb2e652664, State: Initialized, Role: FOLLOWER
I20260924 19:26:31.283550 20397 consensus_queue.cc:260] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.283584 20398 raft_consensus.cc:348] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.283798 20398 raft_consensus.cc:374] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:31.283875 20398 raft_consensus.cc:729] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ad561caa3847fe8aec9a16dfd573bc, State: Initialized, Role: FOLLOWER
I20260924 19:26:31.283886 20397 ts_tablet_manager.cc:1428] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:31.283998 20245 heartbeater.cc:503] Master 127.18.229.190:38567 was elected leader, sending a full tablet report...
I20260924 19:26:31.284022 20398 consensus_queue.cc:260] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.284337 20398 ts_tablet_manager.cc:1428] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:26:31.284343 20379 heartbeater.cc:503] Master 127.18.229.190:38567 was elected leader, sending a full tablet report...
W20260924 19:26:31.337180 20246 tablet.cc:2369] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:31.395958 20404 raft_consensus.cc:482] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:31.396103 20404 raft_consensus.cc:504] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.396544 20404 leader_election.cc:288] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f5560f12fec45778209e971b30020f5 (127.18.229.129:45881), 837e64f2cb1b4ebab19ab4fb2e652664 (127.18.229.130:44007)
I20260924 19:26:31.399276 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.396823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:47929 (local address 127.18.229.129:45881)
0924 19:26:31.396891 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.396893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.396903 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:31.396931 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.396932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.396940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.397008 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:31.397171 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.397631 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.398563 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.398920 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.399004 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.399014 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.399176 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.399218 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:26:31.399272 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.396750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45881 (local address 127.18.229.131:47929)
0924 19:26:31.396809 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.396821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.396872 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.397019 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.397020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.397022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:31.397100 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.397102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.397712 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.397713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.398461 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.398462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.399001 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.399002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.399092 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.399095 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.399158 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.399201 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:26:31.399552 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.396855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:35361 (local address 127.18.229.130:44007)
0924 19:26:31.396902 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.396904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.397146 (+ 242us) server_negotiation.cc:407] Connection header received
0924 19:26:31.397209 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.397210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.397220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.397267 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:31.397415 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.397855 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.398770 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.399148 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.399245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.399253 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.399438 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.399490 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:26:31.399638 20065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af023f4cd6d4a02b24d89d489d8a77b" candidate_uuid: "60ad561caa3847fe8aec9a16dfd573bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5560f12fec45778209e971b30020f5" is_pre_election: true
I20260924 19:26:31.399765 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.396751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:44007 (local address 127.18.229.131:35361)
0924 19:26:31.397063 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:31.397076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:31.397151 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:31.397269 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:31.397270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:31.397271 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:31.397344 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.397346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.397944 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.397945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.398675 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:31.398677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.399250 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.399251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:31.399339 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.399343 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:31.399630 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 19:26:31.399678 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":129,"thread_start_us":130,"threads_started":1}
I20260924 19:26:31.399803 20065 raft_consensus.cc:2454] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ad561caa3847fe8aec9a16dfd573bc in term 0.
I20260924 19:26:31.400081 20268 leader_election.cc:302] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ad561caa3847fe8aec9a16dfd573bc, 8f5560f12fec45778209e971b30020f5; no voters:
I20260924 19:26:31.400192 20199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af023f4cd6d4a02b24d89d489d8a77b" candidate_uuid: "60ad561caa3847fe8aec9a16dfd573bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" is_pre_election: true
I20260924 19:26:31.400239 20404 raft_consensus.cc:2781] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:31.400290 20404 raft_consensus.cc:482] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:31.400342 20404 raft_consensus.cc:3037] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:31.400349 20199 raft_consensus.cc:2454] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ad561caa3847fe8aec9a16dfd573bc in term 0.
I20260924 19:26:31.401085 20404 raft_consensus.cc:504] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.401252 20404 leader_election.cc:288] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [CANDIDATE]: Term 1 election: Requested vote from peers 8f5560f12fec45778209e971b30020f5 (127.18.229.129:45881), 837e64f2cb1b4ebab19ab4fb2e652664 (127.18.229.130:44007)
I20260924 19:26:31.401420 20199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af023f4cd6d4a02b24d89d489d8a77b" candidate_uuid: "60ad561caa3847fe8aec9a16dfd573bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "837e64f2cb1b4ebab19ab4fb2e652664"
I20260924 19:26:31.401417 20065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af023f4cd6d4a02b24d89d489d8a77b" candidate_uuid: "60ad561caa3847fe8aec9a16dfd573bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5560f12fec45778209e971b30020f5"
I20260924 19:26:31.401501 20199 raft_consensus.cc:3037] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:31.401501 20065 raft_consensus.cc:3037] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:31.402379 20199 raft_consensus.cc:2454] T 2af023f4cd6d4a02b24d89d489d8a77b P 837e64f2cb1b4ebab19ab4fb2e652664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ad561caa3847fe8aec9a16dfd573bc in term 1.
I20260924 19:26:31.402379 20065 raft_consensus.cc:2454] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ad561caa3847fe8aec9a16dfd573bc in term 1.
I20260924 19:26:31.402549 20268 leader_election.cc:302] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ad561caa3847fe8aec9a16dfd573bc, 8f5560f12fec45778209e971b30020f5; no voters:
I20260924 19:26:31.402652 20404 raft_consensus.cc:2781] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:31.402792 20404 raft_consensus.cc:686] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [term 1 LEADER]: Becoming Leader. State: Replica: 60ad561caa3847fe8aec9a16dfd573bc, State: Running, Role: LEADER
I20260924 19:26:31.402891 20404 consensus_queue.cc:237] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } }
I20260924 19:26:31.403678 19920 catalog_manager.cc:5495] T 2af023f4cd6d4a02b24d89d489d8a77b P 60ad561caa3847fe8aec9a16dfd573bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ad561caa3847fe8aec9a16dfd573bc (127.18.229.131). New cstate: current_term: 1 leader_uuid: "60ad561caa3847fe8aec9a16dfd573bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ad561caa3847fe8aec9a16dfd573bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 42133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f5560f12fec45778209e971b30020f5" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "837e64f2cb1b4ebab19ab4fb2e652664" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 44007 } health_report { overall_health: UNKNOWN } } }
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:31.420383 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.413196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.18.229.129:45881)
0924 19:26:31.413267 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.413269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.413348 (+ 79us) server_negotiation.cc:407] Connection header received
0924 19:26:31.413412 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.413413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.413422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.413465 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:31.413967 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.414416 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.415077 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.415249 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.415296 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.417600 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:31.417610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:31.417610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:31.417616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:31.419227 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.419511 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.419514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.419514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.419565 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.419742 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.419743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.419743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.419911 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:31.420087 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.420201 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.420281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
W20260924 19:26:31.440924 20112 tablet.cc:2369] T 2af023f4cd6d4a02b24d89d489d8a77b P 8f5560f12fec45778209e971b30020f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:31.442575 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.435563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.18.229.129:45881)
0924 19:26:31.435642 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.435644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.435727 (+ 83us) server_negotiation.cc:407] Connection header received
0924 19:26:31.435822 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.435823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.435832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.435878 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:31.436587 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.437058 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.437874 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.438072 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.438145 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.440406 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:31.440407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:31.440408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:31.440414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:31.441666 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.441960 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.441961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.441962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.442016 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.442173 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.442173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.442174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.442255 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:31.442327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.442458 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.442515 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:31.463641 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:31.457210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.18.229.129:45881)
0924 19:26:31.457259 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:31.457260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:31.457341 (+ 81us) server_negotiation.cc:407] Connection header received
0924 19:26:31.457380 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:31.457381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:31.457389 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:31.457453 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:31.457927 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.458369 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.459040 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:31.459211 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:31.459258 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:31.461414 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:31.461415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:31.461416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:31.461422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:31.462561 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.462902 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.462903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.462904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.462959 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:31.463160 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:31.463161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:31.463161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:31.463257 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:31.463355 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:31.463468 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 19:26:31.463540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:26:31.464061 20025 tablet_service.cc:2908] Rejecting scan request for tablet 2af023f4cd6d4a02b24d89d489d8a77b: Uninitialized: clean time has not yet been initialized
I20260924 19:26:31.464931 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 19982
I20260924 19:26:31.470005 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 20116
I20260924 19:26:31.474291 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 20250
I20260924 19:26:31.478750 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 19888
2026-09-24T19:26:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2017 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[20420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[20420](info): set up 2 sockets
Sep 24 19:26:31 dist-test-slave-r8cc krb5kdc[20420](info): commencing operation
krb5kdc: starting...
W20260924 19:26:32.499800 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:32 dist-test-slave-r8cc krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:32.614408 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42221
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:32.695783 20433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:32.696010 20433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:32.696033 20433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:32.697244 20433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:32.697288 20433 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:26:32.697301 20433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:32.697315 20433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:32.697377 20433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:32.697393 20433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:32.698928 20433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:32.699204 20433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:32.699450 20433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:32.704453 20438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:32.704447 20441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:32.704447 20439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:32.704499 20433 server_base.cc:1034] running on GCE node
I20260924 19:26:32.704981 20433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:32.705287 20433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:32.706487 20433 hybrid_clock.cc:648] HybridClock initialized: now 1790277992706466 us; error 37 us; skew 500 ppm
Sep 24 19:26:32 dist-test-slave-r8cc krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277992, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:32.714210 20433 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:32.714761 20433 webserver.cc:460] Webserver started at http://127.18.229.190:38813/ using document root <none> and password file <none>
I20260924 19:26:32.715010 20433 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:32.715080 20433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:32.715224 20433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:32.716379 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754"
format_stamp: "Formatted at 2026-09-24 19:26:32 on dist-test-slave-r8cc"
server_key: "6acd2c577c6fd0d1851432b7804d1d7d"
server_key_iv: "e36b33b5e4dd2725a2651548301ee163"
server_key_version: "encryptionkey@0"
I20260924 19:26:32.716745 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754"
format_stamp: "Formatted at 2026-09-24 19:26:32 on dist-test-slave-r8cc"
server_key: "6acd2c577c6fd0d1851432b7804d1d7d"
server_key_iv: "e36b33b5e4dd2725a2651548301ee163"
server_key_version: "encryptionkey@0"
I20260924 19:26:32.718093 20433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:32.718797 20448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:32.718964 20433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:32.719065 20433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754"
format_stamp: "Formatted at 2026-09-24 19:26:32 on dist-test-slave-r8cc"
server_key: "6acd2c577c6fd0d1851432b7804d1d7d"
server_key_iv: "e36b33b5e4dd2725a2651548301ee163"
server_key_version: "encryptionkey@0"
I20260924 19:26:32.719142 20433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:32.726274 20433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:32.728533 20433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:32.728690 20433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:32.733388 20433 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42221
I20260924 19:26:32.733440 20510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42221 every 8 connection(s)
I20260924 19:26:32.733759 20433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:32.734336 20511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:32.736639 20511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: Bootstrap starting.
I20260924 19:26:32.737217 20511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:32.737507 20511 log.cc:824] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:32.738173 20511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: No bootstrap required, opened a new log
I20260924 19:26:32.738680 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 20433
I20260924 19:26:32.738759 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:32.738953 19350 external_mini_cluster.cc:1428] Setting key 40e7067d5645fafbaf3e189daa673757
I20260924 19:26:32.740696 20511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42221 } }
I20260924 19:26:32.740854 20511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:32.740917 20511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2c4feb1a764e06aea1b6b3cbbd0754, State: Initialized, Role: FOLLOWER
I20260924 19:26:32.741110 20511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42221 } }
I20260924 19:26:32.741207 20511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:32.741246 20511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:32.741297 20511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:32.741824 20511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42221 } }
I20260924 19:26:32.741950 20511 leader_election.cc:302] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f2c4feb1a764e06aea1b6b3cbbd0754; no voters:
I20260924 19:26:32.742126 20511 leader_election.cc:288] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:32.742168 20516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:32.742344 20516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [term 1 LEADER]: Becoming Leader. State: Replica: 2f2c4feb1a764e06aea1b6b3cbbd0754, State: Running, Role: LEADER
I20260924 19:26:32.742381 20511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:32.742506 20516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42221 } }
I20260924 19:26:32.743083 20517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42221 } } }
I20260924 19:26:32.743098 20518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f2c4feb1a764e06aea1b6b3cbbd0754. Latest consensus state: current_term: 1 leader_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f2c4feb1a764e06aea1b6b3cbbd0754" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42221 } } }
I20260924 19:26:32.743317 20517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:32.743326 20518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:32.743646 20525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:32.744185 20525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:32.746591 20525 catalog_manager.cc:1347] Generated new cluster ID: 65655b9b773d4e0ba4955cad16ef9812
I20260924 19:26:32.746696 20525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:32 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:32.751710 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:32.739515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.18.229.190:42221)
0924 19:26:32.739713 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:32.739721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:32.739745 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:26:32.740676 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:32.740682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:32.740731 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:32.740875 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:32.741571 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.742917 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.743982 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.744759 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.744880 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:32.747660 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:32.747664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:32.747664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:32.747677 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:32.749879 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:32.750287 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:32.750291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:32.750291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:32.750453 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:32.750680 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:32.750681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:32.750681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:32.751063 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:32.751232 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:32.751324 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 19:26:32.751477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260924 19:26:32.758721 20525 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:32.759271 20525 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:32.766757 20525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: Generated new TSK 0
I20260924 19:26:32.766976 20525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:32.797252 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42221
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:32.889237 20537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:32.889453 20537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:32.889478 20537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:32.890664 20537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:32.890707 20537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:32.890726 20537 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:26:32.890743 20537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:32.892233 20537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:42221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:32.892416 20537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:32.892597 20537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:32.897491 20545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:32.897538 20543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:32.897647 20537 server_base.cc:1034] running on GCE node
W20260924 19:26:32.897500 20542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:32.897913 20537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:32.898188 20537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:32.899360 20537 hybrid_clock.cc:648] HybridClock initialized: now 1790277992899338 us; error 39 us; skew 500 ppm
Sep 24 19:26:32 dist-test-slave-r8cc krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277992, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:32.902055 20537 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:32.902504 20537 webserver.cc:460] Webserver started at http://127.18.229.129:45065/ using document root <none> and password file <none>
I20260924 19:26:32.902732 20537 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:32.902810 20537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:32.902968 20537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:32.904026 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "895e3b3b70e543c0ad38c24969dcdee2"
format_stamp: "Formatted at 2026-09-24 19:26:32 on dist-test-slave-r8cc"
server_key: "308eee59d590d64703571610d8764620"
server_key_iv: "362db72eaba916ec1da188ddf17926c7"
server_key_version: "encryptionkey@0"
I20260924 19:26:32.904457 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "895e3b3b70e543c0ad38c24969dcdee2"
format_stamp: "Formatted at 2026-09-24 19:26:32 on dist-test-slave-r8cc"
server_key: "308eee59d590d64703571610d8764620"
server_key_iv: "362db72eaba916ec1da188ddf17926c7"
server_key_version: "encryptionkey@0"
I20260924 19:26:32.905767 20537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:32.906507 20552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:32.906675 20537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:32.906754 20537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "895e3b3b70e543c0ad38c24969dcdee2"
format_stamp: "Formatted at 2026-09-24 19:26:32 on dist-test-slave-r8cc"
server_key: "308eee59d590d64703571610d8764620"
server_key_iv: "362db72eaba916ec1da188ddf17926c7"
server_key_version: "encryptionkey@0"
I20260924 19:26:32.906806 20537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:32.917302 20537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:32.919050 20537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:32.919160 20537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:32.919602 20537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:32.919642 20537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:32.919664 20537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:32.919708 20537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:32.926553 20537 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:34667
I20260924 19:26:32.926612 20667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:34667 every 8 connection(s)
Sep 24 19:26:32 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277992, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:32.926949 20537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:32.930056 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:32.920140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:47125 (local address 127.18.229.190:42221)
0924 19:26:32.920181 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:32.920183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:32.920422 (+ 239us) server_negotiation.cc:407] Connection header received
0924 19:26:32.920990 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:32.920992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:32.921012 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:32.921087 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:32.922058 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.922673 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.924222 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.924599 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.924681 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:32.927368 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:32.927369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:32.927369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:32.927380 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:32.928799 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:32.929201 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:32.929202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:32.929202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:32.929254 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:32.929481 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:32.929481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:32.929482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:32.929582 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:32.929633 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:32.929909 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:26:32.929994 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:26:32.930248 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:32.920141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.18.229.129:47125)
0924 19:26:32.920306 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:32.920340 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:32.920838 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:32.921182 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:32.921188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:32.921589 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:32.921972 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:32.921980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.922775 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.922776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:32.924143 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:32.924146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.924697 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.924698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:32.924794 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:32.925361 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:32.925373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:32.927276 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:32.928882 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:32.928888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:32.928890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:32.929129 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:32.929314 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:32.929315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:32.929315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:32.929427 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:32.929664 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:32.929670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:32.929791 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 19:26:32.929885 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:26:32.930009 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260924 19:26:32.930775 20668 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42221
I20260924 19:26:32.930857 20668 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:32.931044 20668 heartbeater.cc:511] Master 127.18.229.190:42221 requested a full tablet report, sending...
I20260924 19:26:32.931594 20464 ts_manager.cc:194] Registered new tserver with Master: 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667)
I20260924 19:26:32.932194 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 20537
I20260924 19:26:32.932315 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:32.932489 19350 external_mini_cluster.cc:1428] Setting key 1aa4c473ffbafc6d297d3c3af25c6c0a
I20260924 19:26:32.932569 20464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:47125
I20260924 19:26:32.937726 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:32.931831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.18.229.129:43323)
0924 19:26:32.931947 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:32.931958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:32.932014 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:32.932214 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:32.932215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:32.932412 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:32.932470 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:32.932472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.933387 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.933388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:32.933890 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:32.933892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.934356 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.934357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:32.934412 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:32.934874 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:32.934879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:32.935169 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:32.936686 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:32.936687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:32.936688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:32.936926 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:32.937111 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:32.937112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:32.937112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:32.937181 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:32.937396 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:32.937397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:32.937458 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 19:26:32.937530 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:26:32.937648 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260924 19:26:32.937775 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:32.931894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:43323 (local address 127.18.229.190:42221)
0924 19:26:32.931941 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:32.931943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:32.932001 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:26:32.932074 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:32.932074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:32.932084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:32.932172 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:32.932561 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.933297 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.933970 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:32.934282 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:32.934355 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:32.935251 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:32.935253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:32.935253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:32.935261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:32.936600 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:32.936997 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:32.936998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:32.936998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:32.937042 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:32.937242 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:32.937242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:32.937242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:32.937331 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:32.937380 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:32.937566 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 19:26:32.937651 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:32.958989 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42221
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:33.040819 20677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:33.041038 20677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:33.041066 20677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:33.042563 20677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:33.042616 20677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:33.042639 20677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:26:33.042656 20677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:33.044348 20677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:42221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:33.044597 20677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:33.044837 20677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:33.049602 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:33.049726 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:33.049723 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:33.049945 20677 server_base.cc:1034] running on GCE node
I20260924 19:26:33.050161 20677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:33.050420 20677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:33.051573 20677 hybrid_clock.cc:648] HybridClock initialized: now 1790277993051569 us; error 36 us; skew 500 ppm
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:33.054176 20677 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:33.054642 20677 webserver.cc:460] Webserver started at http://127.18.229.130:34603/ using document root <none> and password file <none>
I20260924 19:26:33.054872 20677 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:33.054946 20677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:33.055104 20677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:33.056339 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "3d0b4232f5554028b9bfe4d261b2008b"
format_stamp: "Formatted at 2026-09-24 19:26:33 on dist-test-slave-r8cc"
server_key: "77ffed7b04540e695200e6ddb13b3d2f"
server_key_iv: "22639d49ffc3977b5722cc3744a5d350"
server_key_version: "encryptionkey@0"
I20260924 19:26:33.056736 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "3d0b4232f5554028b9bfe4d261b2008b"
format_stamp: "Formatted at 2026-09-24 19:26:33 on dist-test-slave-r8cc"
server_key: "77ffed7b04540e695200e6ddb13b3d2f"
server_key_iv: "22639d49ffc3977b5722cc3744a5d350"
server_key_version: "encryptionkey@0"
I20260924 19:26:33.058094 20677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:33.058815 20692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.058986 20677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:33.059091 20677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "3d0b4232f5554028b9bfe4d261b2008b"
format_stamp: "Formatted at 2026-09-24 19:26:33 on dist-test-slave-r8cc"
server_key: "77ffed7b04540e695200e6ddb13b3d2f"
server_key_iv: "22639d49ffc3977b5722cc3744a5d350"
server_key_version: "encryptionkey@0"
I20260924 19:26:33.059165 20677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:33.079896 20677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:33.082397 20677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:33.082566 20677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:33.083215 20677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:33.083264 20677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.083302 20677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:33.083316 20677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:33.091037 20677 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:45329
I20260924 19:26:33.091236 20807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:45329 every 8 connection(s)
I20260924 19:26:33.091476 20677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:33.093807 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 20677
I20260924 19:26:33.093887 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:33.094091 19350 external_mini_cluster.cc:1428] Setting key 5dd5c7512e7e2443782accf79b111705
I20260924 19:26:33.094166 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.083559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57599 (local address 127.18.229.190:42221)
0924 19:26:33.083629 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.083632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.083924 (+ 292us) server_negotiation.cc:407] Connection header received
0924 19:26:33.084900 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.084902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.084916 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.084990 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:33.085987 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.086557 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.088251 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.088467 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.088542 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.091060 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:33.091062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:33.091062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:33.091070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:33.092764 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.093174 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.093175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.093176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.093236 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.093465 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.093465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.093466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.093575 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:33.093685 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.093967 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.094065 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:26:33.094343 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.083567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.18.229.130:57599)
0924 19:26:33.083747 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.083801 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.084722 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.085099 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.085105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.085543 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:33.085870 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.085876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.086642 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.086643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.088151 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.088154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.088539 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.088541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.088712 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.089224 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:33.089236 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:33.090958 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:33.092850 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.092855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.092856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.093070 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.093304 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.093304 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.093304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.093398 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.093670 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:33.093675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:33.093791 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.093933 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.094106 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260924 19:26:33.094820 20808 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42221
I20260924 19:26:33.094916 20808 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:33.095146 20808 heartbeater.cc:511] Master 127.18.229.190:42221 requested a full tablet report, sending...
I20260924 19:26:33.095491 20464 ts_manager.cc:194] Registered new tserver with Master: 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329)
I20260924 19:26:33.095959 20464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:57599
I20260924 19:26:33.102372 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.095871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:33385 (local address 127.18.229.190:42221)
0924 19:26:33.095907 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.095909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.096014 (+ 105us) server_negotiation.cc:407] Connection header received
0924 19:26:33.096139 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.096140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.096153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.096186 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:33.096614 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.097348 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.098470 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.098676 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.098728 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.099850 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:33.099851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:33.099852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:33.099857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:33.101292 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.101668 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.101668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.101669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.101715 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.101908 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.101908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.101909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.102033 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:33.102086 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.102257 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.102314 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:26:33.102473 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.095809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.18.229.130:33385)
0924 19:26:33.095950 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.095961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.096043 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.096247 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.096248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.096464 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:33.096529 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.096531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.097458 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.097459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.098381 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.098383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.098752 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.098753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.098834 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.099272 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:33.099278 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:33.099687 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:33.101373 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.101374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.101375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.101587 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.101776 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.101777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.101777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.101840 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.102104 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:33.102105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:33.102154 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.102243 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.102359 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:33.118374 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42221
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:33.198312 20817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:33.198566 20817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:33.198608 20817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:33.199852 20817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:33.199924 20817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:33.199960 20817 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:26:33.200001 20817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:33.201445 20817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:42221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:33.201727 20817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:33.201961 20817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:33.206959 20823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:33.207032 20822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:33.207085 20817 server_base.cc:1034] running on GCE node
W20260924 19:26:33.207054 20825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:33.207377 20817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:33.207626 20817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:33.208765 20817 hybrid_clock.cc:648] HybridClock initialized: now 1790277993208751 us; error 29 us; skew 500 ppm
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:33.211189 20817 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:33.211694 20817 webserver.cc:460] Webserver started at http://127.18.229.131:35887/ using document root <none> and password file <none>
I20260924 19:26:33.211926 20817 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:33.211970 20817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:33.212163 20817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:33.213212 20817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "e0db79d170da49a0a082957f9b691de0"
format_stamp: "Formatted at 2026-09-24 19:26:33 on dist-test-slave-r8cc"
server_key: "1b927f63d6f9e06dbbb9ad217f8df537"
server_key_iv: "61021334827e014119c9315de60bf410"
server_key_version: "encryptionkey@0"
I20260924 19:26:33.213555 20817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "e0db79d170da49a0a082957f9b691de0"
format_stamp: "Formatted at 2026-09-24 19:26:33 on dist-test-slave-r8cc"
server_key: "1b927f63d6f9e06dbbb9ad217f8df537"
server_key_iv: "61021334827e014119c9315de60bf410"
server_key_version: "encryptionkey@0"
I20260924 19:26:33.214809 20817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:26:33.215494 20832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.215664 20817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:33.215758 20817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "e0db79d170da49a0a082957f9b691de0"
format_stamp: "Formatted at 2026-09-24 19:26:33 on dist-test-slave-r8cc"
server_key: "1b927f63d6f9e06dbbb9ad217f8df537"
server_key_iv: "61021334827e014119c9315de60bf410"
server_key_version: "encryptionkey@0"
I20260924 19:26:33.215838 20817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:33.231400 20817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:33.234352 20817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:33.234519 20817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:33.235196 20817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:33.235270 20817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.235306 20817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:33.235332 20817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.243245 20817 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:38715
I20260924 19:26:33.243305 20947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:38715 every 8 connection(s)
I20260924 19:26:33.243736 20817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:33.247282 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.235888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:50215 (local address 127.18.229.190:42221)
0924 19:26:33.235938 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.235941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.236315 (+ 374us) server_negotiation.cc:407] Connection header received
0924 19:26:33.236987 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.237005 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.237019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.237093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:33.238119 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.239794 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.242044 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.242370 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.242432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.245087 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:33.245088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:33.245089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:33.245095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:33.246211 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.246597 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.246597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.246597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.246645 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.246796 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.246797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.246797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.246885 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:33.246941 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.247162 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.247222 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:26:33.247426 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.235943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.18.229.131:50215)
0924 19:26:33.236152 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.236203 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.236882 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.237241 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.237247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.237664 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:33.237963 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.237971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.239957 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.239958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.241967 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.241970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.242484 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.242486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.242629 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.243289 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:33.243299 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:33.245006 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:33.246287 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.246292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.246295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.246522 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.246682 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.246683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.246683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.246761 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.246945 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:33.246947 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:33.247047 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.247118 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.247227 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260924 19:26:33.247893 20948 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42221
I20260924 19:26:33.247977 20948 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:33.248193 20948 heartbeater.cc:511] Master 127.18.229.190:42221 requested a full tablet report, sending...
I20260924 19:26:33.248522 20464 ts_manager.cc:194] Registered new tserver with Master: e0db79d170da49a0a082957f9b691de0 (127.18.229.131:38715)
I20260924 19:26:33.248894 20464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:50215
I20260924 19:26:33.253104 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 20817
I20260924 19:26:33.253175 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:33.253340 19350 external_mini_cluster.cc:1428] Setting key 31b85549fcd3ca479193870b55a7df1d
I20260924 19:26:33.254086 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.248944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.18.229.131:46455)
0924 19:26:33.249073 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.249083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.249142 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.249257 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.249258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.249461 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:33.249515 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.249516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.250104 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.250105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.250576 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.250577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.250865 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.250865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.250920 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.251342 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:33.251347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:33.251634 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:33.253003 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.253004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.253004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.253182 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.253414 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.253415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.253415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.253472 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.253764 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:33.253765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:33.253811 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.253881 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.253973 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260924 19:26:33.254096 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.249004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:46455 (local address 127.18.229.190:42221)
0924 19:26:33.249037 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.249039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.249152 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:26:33.249186 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.249187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.249197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.249246 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:33.249585 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.250024 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.250640 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.250806 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.250851 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.251710 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:33.251711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:33.251712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:33.251718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:33.252886 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.253278 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.253280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.253280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.253341 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.253560 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.253561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.253562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.253697 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:33.253763 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.253923 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.253980 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:26:33.254488 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:33.275647 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.269366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.18.229.190:42221)
0924 19:26:33.269426 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.269428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.269437 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:33.269459 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.269460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.269469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.269517 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:33.270017 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.270484 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.271145 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.271319 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.271370 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.273517 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:33.273518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:33.273519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:33.273525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:33.274658 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.274959 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.274960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.274960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.275003 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.275158 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.275159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.275159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.275261 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:33.275330 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.275457 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.275516 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:26:33.277334 20464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:33.278863 20464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:33.285910 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.281673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.18.229.131:38715)
0924 19:26:33.281849 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.281854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.281864 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:33.281922 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.281925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.281941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.282026 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.282216 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.283344 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.284638 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.285386 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.285477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.285540 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.285753 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.285807 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260924 19:26:33.285905 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.281585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:38715 (local address 127.0.0.1:34296)
0924 19:26:33.281701 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.281712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.281755 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.282094 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.282095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.282097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.282127 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.282128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.283439 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.283440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.284537 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.284540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.285579 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.285580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.285661 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.285664 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.285737 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.285807 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":55,"thread_start_us":77,"threads_started":1}
I20260924 19:26:33.285915 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.281367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:34667 (local address 127.0.0.1:60164)
0924 19:26:33.281496 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.281539 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.281612 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.281966 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.281969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.281972 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.282041 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.282043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.283406 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.283406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.284669 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.284672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.285538 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.285539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.285631 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.285645 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.285743 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.285817 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":60,"thread_start_us":59,"threads_started":1}
I20260924 19:26:33.286329 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.281489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.18.229.129:34667)
0924 19:26:33.281752 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.281758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.281767 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:33.281816 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.281818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.281834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.282119 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.282186 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.283316 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.284754 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.285456 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.285522 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.285569 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.286190 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.286240 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260924 19:26:33.286641 20882 tablet_service.cc:1467] Processing CreateTablet for tablet fecf55c1269a42ffa3c364be23c7de8a (DEFAULT_TABLE table=test-table [id=662177bc983a402d92f87048d0c6dd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:33.287117 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.281748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:45329 (local address 127.0.0.1:60318)
0924 19:26:33.282164 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.282173 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.282217 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.283781 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.283782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.283785 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.283826 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.283827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.285061 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.285062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.285935 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.285937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.286869 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.286870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.286931 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.286936 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.286999 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.287041 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":51,"threads_started":1}
I20260924 19:26:33.287156 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.281955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.18.229.130:45329)
0924 19:26:33.283622 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.283627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.283636 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:33.283671 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.283673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.283687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.283764 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.283926 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.284967 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.286034 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.286787 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.286860 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.286910 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.287020 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.287059 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1628,"thread_start_us":70,"threads_started":1}
I20260924 19:26:33.287034 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf55c1269a42ffa3c364be23c7de8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.287227 20602 tablet_service.cc:1467] Processing CreateTablet for tablet fecf55c1269a42ffa3c364be23c7de8a (DEFAULT_TABLE table=test-table [id=662177bc983a402d92f87048d0c6dd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:33.287503 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf55c1269a42ffa3c364be23c7de8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.289914 20971 tablet_bootstrap.cc:492] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Bootstrap starting.
I20260924 19:26:33.290297 20742 tablet_service.cc:1467] Processing CreateTablet for tablet fecf55c1269a42ffa3c364be23c7de8a (DEFAULT_TABLE table=test-table [id=662177bc983a402d92f87048d0c6dd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:33.290588 20972 tablet_bootstrap.cc:492] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Bootstrap starting.
I20260924 19:26:33.290657 20742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf55c1269a42ffa3c364be23c7de8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.290967 20971 tablet_bootstrap.cc:654] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.291225 20972 tablet_bootstrap.cc:654] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.291297 20971 log.cc:824] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:33.291577 20972 log.cc:824] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:33.292049 20971 tablet_bootstrap.cc:492] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: No bootstrap required, opened a new log
I20260924 19:26:33.292160 20971 ts_tablet_manager.cc:1397] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:33.292423 20972 tablet_bootstrap.cc:492] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: No bootstrap required, opened a new log
I20260924 19:26:33.292490 20972 ts_tablet_manager.cc:1397] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:33.293946 20975 tablet_bootstrap.cc:492] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Bootstrap starting.
I20260924 19:26:33.294176 20971 raft_consensus.cc:348] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.294345 20971 raft_consensus.cc:374] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.294382 20971 raft_consensus.cc:729] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0db79d170da49a0a082957f9b691de0, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.294416 20972 raft_consensus.cc:348] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.294556 20972 raft_consensus.cc:374] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.294608 20972 raft_consensus.cc:729] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895e3b3b70e543c0ad38c24969dcdee2, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.294628 20975 tablet_bootstrap.cc:654] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.294749 20972 consensus_queue.cc:260] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.294785 20971 consensus_queue.cc:260] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.294996 20975 log.cc:824] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:33.295018 20971 ts_tablet_manager.cc:1428] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:33.295055 20948 heartbeater.cc:503] Master 127.18.229.190:42221 was elected leader, sending a full tablet report...
I20260924 19:26:33.295018 20972 ts_tablet_manager.cc:1428] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:26:33.295414 20668 heartbeater.cc:503] Master 127.18.229.190:42221 was elected leader, sending a full tablet report...
I20260924 19:26:33.295811 20975 tablet_bootstrap.cc:492] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: No bootstrap required, opened a new log
I20260924 19:26:33.295881 20975 ts_tablet_manager.cc:1397] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:33.297403 20975 raft_consensus.cc:348] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.297565 20975 raft_consensus.cc:374] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.297608 20975 raft_consensus.cc:729] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b4232f5554028b9bfe4d261b2008b, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.297712 20975 consensus_queue.cc:260] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.297963 20975 ts_tablet_manager.cc:1428] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:26:33.298004 20808 heartbeater.cc:503] Master 127.18.229.190:42221 was elected leader, sending a full tablet report...
W20260924 19:26:33.342043 20809 tablet.cc:2369] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:26:33.427738 20669 tablet.cc:2369] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:26:33.494445 20949 tablet.cc:2369] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:33.502418 20979 raft_consensus.cc:482] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:33.502523 20979 raft_consensus.cc:504] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.502815 20979 leader_election.cc:288] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667), e0db79d170da49a0a082957f9b691de0 (127.18.229.131:38715)
I20260924 19:26:33.505816 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.503093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:40523 (local address 127.18.229.129:34667)
0924 19:26:33.503156 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.503159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.503186 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:26:33.503275 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.503276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.503287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.503348 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.503539 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.504153 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.505130 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.505491 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.505542 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.505549 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.505708 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.505751 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:26:33.505816 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.503026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:34667 (local address 127.18.229.130:40523)
0924 19:26:33.503100 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.503128 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.503206 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.503383 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.503385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.503388 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.503468 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.503470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.504253 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.504254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.505041 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.505043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.505561 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.505562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.505639 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.505642 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.505694 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.505723 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":2}
I20260924 19:26:33.505887 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.503238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57015 (local address 127.18.229.131:38715)
0924 19:26:33.503280 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.503282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.503379 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:26:33.503435 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.503436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.503448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.503515 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.503687 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.504257 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.505181 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.505512 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.505599 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.505606 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.505777 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.505813 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:26:33.505939 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.503154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:38715 (local address 127.18.229.130:57015)
0924 19:26:33.503315 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.503324 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.503370 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.503539 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.503540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.503541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.503613 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.503614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.504329 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.504329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.505059 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.505060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.505619 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.505619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.505696 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.505701 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.505755 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.505790 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260924 19:26:33.506214 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf55c1269a42ffa3c364be23c7de8a" candidate_uuid: "3d0b4232f5554028b9bfe4d261b2008b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895e3b3b70e543c0ad38c24969dcdee2" is_pre_election: true
I20260924 19:26:33.506214 20895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf55c1269a42ffa3c364be23c7de8a" candidate_uuid: "3d0b4232f5554028b9bfe4d261b2008b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0db79d170da49a0a082957f9b691de0" is_pre_election: true
I20260924 19:26:33.506358 20895 raft_consensus.cc:2454] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0b4232f5554028b9bfe4d261b2008b in term 0.
I20260924 19:26:33.506358 20622 raft_consensus.cc:2454] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0b4232f5554028b9bfe4d261b2008b in term 0.
I20260924 19:26:33.506598 20696 leader_election.cc:302] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0b4232f5554028b9bfe4d261b2008b, e0db79d170da49a0a082957f9b691de0; no voters:
I20260924 19:26:33.506716 20979 raft_consensus.cc:2781] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:33.506803 20979 raft_consensus.cc:482] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:33.506857 20979 raft_consensus.cc:3037] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:33.507731 20979 raft_consensus.cc:504] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.507861 20979 leader_election.cc:288] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [CANDIDATE]: Term 1 election: Requested vote from peers 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667), e0db79d170da49a0a082957f9b691de0 (127.18.229.131:38715)
I20260924 19:26:33.508023 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf55c1269a42ffa3c364be23c7de8a" candidate_uuid: "3d0b4232f5554028b9bfe4d261b2008b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895e3b3b70e543c0ad38c24969dcdee2"
I20260924 19:26:33.508054 20895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf55c1269a42ffa3c364be23c7de8a" candidate_uuid: "3d0b4232f5554028b9bfe4d261b2008b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0db79d170da49a0a082957f9b691de0"
I20260924 19:26:33.508124 20622 raft_consensus.cc:3037] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:33.508152 20895 raft_consensus.cc:3037] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:33.508988 20895 raft_consensus.cc:2454] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0b4232f5554028b9bfe4d261b2008b in term 1.
I20260924 19:26:33.508988 20622 raft_consensus.cc:2454] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0b4232f5554028b9bfe4d261b2008b in term 1.
I20260924 19:26:33.509117 20696 leader_election.cc:302] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0b4232f5554028b9bfe4d261b2008b, 895e3b3b70e543c0ad38c24969dcdee2; no voters:
I20260924 19:26:33.509198 20979 raft_consensus.cc:2781] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:33.509339 20979 raft_consensus.cc:686] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 1 LEADER]: Becoming Leader. State: Replica: 3d0b4232f5554028b9bfe4d261b2008b, State: Running, Role: LEADER
I20260924 19:26:33.509441 20979 consensus_queue.cc:237] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } }
I20260924 19:26:33.510218 20464 catalog_manager.cc:5495] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130). New cstate: current_term: 1 leader_uuid: "3d0b4232f5554028b9bfe4d261b2008b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } health_report { overall_health: HEALTHY } } }
I20260924 19:26:33.569923 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.567477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.18.229.130:45329)
0924 19:26:33.567534 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.567536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.567550 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:26:33.567581 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.567582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.567597 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.567655 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:33.567803 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.568326 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.569113 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.569356 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.569415 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.569610 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:33.569697 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:33.569754 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.569796 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.569813 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:26:33.571676 20895 raft_consensus.cc:1261] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Refusing update from remote peer 3d0b4232f5554028b9bfe4d261b2008b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:33.571676 20622 raft_consensus.cc:1261] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Refusing update from remote peer 3d0b4232f5554028b9bfe4d261b2008b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:33.572083 20981 consensus_queue.cc:1035] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [LEADER]: Connected to new peer: Peer: permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:33.572269 20979 consensus_queue.cc:1035] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:33.574759 20986 mvcc.cc:205] Tried to move back new op lower bound from 7332978661666684928 to 7332978661415272448. Current Snapshot: MvccSnapshot[applied={T|T < 7332978661666684928}]
I20260924 19:26:33.574915 20985 mvcc.cc:205] Tried to move back new op lower bound from 7332978661666684928 to 7332978661415272448. Current Snapshot: MvccSnapshot[applied={T|T < 7332978661666684928}]
I20260924 19:26:33.577570 20464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:33.577669 20464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:33.578442 20464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: Sending DeleteTablet for 3 replicas of tablet fecf55c1269a42ffa3c364be23c7de8a
I20260924 19:26:33.578771 20602 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf55c1269a42ffa3c364be23c7de8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:33 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:60164
I20260924 19:26:33.578809 20882 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf55c1269a42ffa3c364be23c7de8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:33 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:34296
I20260924 19:26:33.579006 20464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:26:33.579061 20742 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf55c1269a42ffa3c364be23c7de8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:33 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:60318
W20260924 19:26:33.579155 20464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:33.579258 20996 tablet_replica.cc:330] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: stopping tablet replica
I20260924 19:26:33.579347 20996 raft_consensus.cc:2229] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:33.579442 20996 raft_consensus.cc:2258] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:33.579581 20994 tablet_replica.cc:330] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: stopping tablet replica
I20260924 19:26:33.579649 20995 tablet_replica.cc:330] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: stopping tablet replica
I20260924 19:26:33.579679 20994 raft_consensus.cc:2229] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:33.579718 20995 raft_consensus.cc:2229] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:33.579764 20994 raft_consensus.cc:2258] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:33.579792 20995 raft_consensus.cc:2258] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:33.580330 20995 ts_tablet_manager.cc:1905] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:33.580377 20996 ts_tablet_manager.cc:1905] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:33.581037 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 79126dab226f4b2cb7c5e970cb6abfad (DEFAULT_TABLE table=test-table [id=f01a87d5ab1d46d1b978bcd2b90a79a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:33.581147 20742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79126dab226f4b2cb7c5e970cb6abfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.581964 20882 tablet_service.cc:1467] Processing CreateTablet for tablet 79126dab226f4b2cb7c5e970cb6abfad (DEFAULT_TABLE table=test-table [id=f01a87d5ab1d46d1b978bcd2b90a79a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:33.582057 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79126dab226f4b2cb7c5e970cb6abfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.582064 20995 ts_tablet_manager.cc:1918] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:33.582154 20995 log.cc:1196] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/fecf55c1269a42ffa3c364be23c7de8a
I20260924 19:26:33.582423 20995 ts_tablet_manager.cc:1939] T fecf55c1269a42ffa3c364be23c7de8a P e0db79d170da49a0a082957f9b691de0: Deleting consensus metadata
I20260924 19:26:33.582726 20975 tablet_bootstrap.cc:492] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Bootstrap starting.
I20260924 19:26:33.582792 20452 catalog_manager.cc:4841] TS e0db79d170da49a0a082957f9b691de0 (127.18.229.131:38715): tablet fecf55c1269a42ffa3c364be23c7de8a (table test-table [id=662177bc983a402d92f87048d0c6dd63]) successfully deleted
I20260924 19:26:33.583118 20975 tablet_bootstrap.cc:654] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.583195 20971 tablet_bootstrap.cc:492] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Bootstrap starting.
I20260924 19:26:33.583343 20602 tablet_service.cc:1467] Processing CreateTablet for tablet 79126dab226f4b2cb7c5e970cb6abfad (DEFAULT_TABLE table=test-table [id=f01a87d5ab1d46d1b978bcd2b90a79a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:33.583447 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79126dab226f4b2cb7c5e970cb6abfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.583531 20996 ts_tablet_manager.cc:1918] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:33.583590 20996 log.cc:1196] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/fecf55c1269a42ffa3c364be23c7de8a
I20260924 19:26:33.583601 20971 tablet_bootstrap.cc:654] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.583832 20996 ts_tablet_manager.cc:1939] T fecf55c1269a42ffa3c364be23c7de8a P 3d0b4232f5554028b9bfe4d261b2008b: Deleting consensus metadata
I20260924 19:26:33.584213 20452 catalog_manager.cc:4841] TS 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329): tablet fecf55c1269a42ffa3c364be23c7de8a (table test-table [id=662177bc983a402d92f87048d0c6dd63]) successfully deleted
I20260924 19:26:33.584488 20971 tablet_bootstrap.cc:492] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: No bootstrap required, opened a new log
I20260924 19:26:33.584535 20975 tablet_bootstrap.cc:492] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: No bootstrap required, opened a new log
I20260924 19:26:33.584529 20971 ts_tablet_manager.cc:1397] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:33.584568 20975 ts_tablet_manager.cc:1397] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:26:33.584692 20971 raft_consensus.cc:348] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.584707 20975 raft_consensus.cc:348] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.584753 20971 raft_consensus.cc:374] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.584760 20975 raft_consensus.cc:374] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.584777 20971 raft_consensus.cc:729] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0db79d170da49a0a082957f9b691de0, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.584784 20975 raft_consensus.cc:729] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b4232f5554028b9bfe4d261b2008b, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.584823 20971 consensus_queue.cc:260] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.584824 20975 consensus_queue.cc:260] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.584899 20971 ts_tablet_manager.cc:1428] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:33.584924 20975 ts_tablet_manager.cc:1428] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.584985 20972 tablet_bootstrap.cc:492] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Bootstrap starting.
I20260924 19:26:33.585386 20972 tablet_bootstrap.cc:654] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.585716 20994 ts_tablet_manager.cc:1905] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:33.586253 20972 tablet_bootstrap.cc:492] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: No bootstrap required, opened a new log
I20260924 19:26:33.586284 20972 ts_tablet_manager.cc:1397] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:33.586387 20972 raft_consensus.cc:348] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.586439 20972 raft_consensus.cc:374] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.586452 20972 raft_consensus.cc:729] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895e3b3b70e543c0ad38c24969dcdee2, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.586510 20972 consensus_queue.cc:260] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.586608 20972 ts_tablet_manager.cc:1428] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.587574 20994 ts_tablet_manager.cc:1918] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:33.587636 20994 log.cc:1196] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/fecf55c1269a42ffa3c364be23c7de8a
I20260924 19:26:33.587878 20994 ts_tablet_manager.cc:1939] T fecf55c1269a42ffa3c364be23c7de8a P 895e3b3b70e543c0ad38c24969dcdee2: Deleting consensus metadata
I20260924 19:26:33.588254 20452 catalog_manager.cc:4841] TS 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667): tablet fecf55c1269a42ffa3c364be23c7de8a (table test-table [id=662177bc983a402d92f87048d0c6dd63]) successfully deleted
I20260924 19:26:33.844857 20977 raft_consensus.cc:482] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:33.844969 20977 raft_consensus.cc:504] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.845368 20977 leader_election.cc:288] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667), 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329)
I20260924 19:26:33.849071 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.845584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:34667 (local address 127.18.229.131:49517)
0924 19:26:33.845907 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.845929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.845984 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.846140 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.846141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.846143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.846207 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.846209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.847160 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.847160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.848177 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.848180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.848820 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.848821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.848889 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.848893 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.848946 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.848985 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":65,"threads_started":1}
I20260924 19:26:33.849095 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.845626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:49517 (local address 127.18.229.129:34667)
0924 19:26:33.845701 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.845703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.845976 (+ 273us) server_negotiation.cc:407] Connection header received
0924 19:26:33.846062 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.846064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.846075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.846134 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.846294 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.847050 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.848283 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.848710 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.848780 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.848788 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.848963 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.849002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:26:33.849315 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79126dab226f4b2cb7c5e970cb6abfad" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895e3b3b70e543c0ad38c24969dcdee2" is_pre_election: true
I20260924 19:26:33.849431 20622 raft_consensus.cc:2454] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 0.
I20260924 19:26:33.849540 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.845847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:34805 (local address 127.18.229.130:45329)
0924 19:26:33.845895 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.845897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.846163 (+ 266us) server_negotiation.cc:407] Connection header received
0924 19:26:33.846361 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.846362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.846377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.846463 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:33.846594 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.847374 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.848624 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.849170 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.849236 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.849243 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.849425 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.849473 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:33.849598 20836 leader_election.cc:302] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895e3b3b70e543c0ad38c24969dcdee2, e0db79d170da49a0a082957f9b691de0; no voters:
I20260924 19:26:33.849725 20977 raft_consensus.cc:2781] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:33.849810 20977 raft_consensus.cc:482] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:33.849810 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.845772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:45329 (local address 127.18.229.131:34805)
0924 19:26:33.846083 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.846093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.846206 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.846463 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.846464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.846465 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:33.846528 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.846531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.847472 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.847472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.848543 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.848545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.849273 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.849274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.849341 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.849344 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.849646 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.849688 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":64,"threads_started":1}
I20260924 19:26:33.849850 20977 raft_consensus.cc:3037] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:33.850112 20762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79126dab226f4b2cb7c5e970cb6abfad" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0b4232f5554028b9bfe4d261b2008b" is_pre_election: true
I20260924 19:26:33.850219 20762 raft_consensus.cc:2454] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 0.
I20260924 19:26:33.850951 20977 raft_consensus.cc:504] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.851091 20977 leader_election.cc:288] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 election: Requested vote from peers 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667), 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329)
I20260924 19:26:33.851256 20762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79126dab226f4b2cb7c5e970cb6abfad" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0b4232f5554028b9bfe4d261b2008b"
I20260924 19:26:33.851245 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79126dab226f4b2cb7c5e970cb6abfad" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895e3b3b70e543c0ad38c24969dcdee2"
I20260924 19:26:33.851322 20762 raft_consensus.cc:3037] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:33.851322 20622 raft_consensus.cc:3037] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:33.852269 20622 raft_consensus.cc:2454] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 1.
I20260924 19:26:33.852269 20762 raft_consensus.cc:2454] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 1.
I20260924 19:26:33.852427 20836 leader_election.cc:302] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895e3b3b70e543c0ad38c24969dcdee2, e0db79d170da49a0a082957f9b691de0; no voters:
I20260924 19:26:33.852509 20977 raft_consensus.cc:2781] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:33.852617 20977 raft_consensus.cc:686] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 1 LEADER]: Becoming Leader. State: Replica: e0db79d170da49a0a082957f9b691de0, State: Running, Role: LEADER
I20260924 19:26:33.852689 20977 consensus_queue.cc:237] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.853381 20465 catalog_manager.cc:5495] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0db79d170da49a0a082957f9b691de0 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } health_report { overall_health: HEALTHY } } }
Sep 24 19:26:33 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277992, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:33.896386 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.888980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42221 (local address 127.0.0.1:41738)
0924 19:26:33.889156 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:33.889165 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:33.889228 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:33.889399 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:33.889399 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:33.889754 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:33.889799 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.889801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.890427 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.890446 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.891394 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:33.891397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.891742 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.891743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:33.891785 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.892408 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:33.892413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:33.893796 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:33.895486 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.895492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.895493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.895675 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.895845 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:33.895846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:33.895847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:33.895896 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:33.896095 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:33.896100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:33.896182 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 19:26:33.896231 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:26:33.896315 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":109,"threads_started":1}
I20260924 19:26:33.896490 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:33.889054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.18.229.190:42221)
0924 19:26:33.889178 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:33.889180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:33.889213 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:26:33.889287 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:33.889288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:33.889297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:33.889364 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:33.889868 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.890365 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.891476 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:33.891679 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:33.891732 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:33.893870 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:33.893871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:33.893871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:33.893877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:33.895420 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.895739 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.895739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.895740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.895789 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:33.895954 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:33.895955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:33.895955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:33.896045 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:33.896091 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:33.896256 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 19:26:33.896364 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260924 19:26:33.898218 20465 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:41738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:26:33.898439 20465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:33.899864 20602 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8b0fa4edea40e29f540ce585be8b72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=867ebfbfbe1640e590ccabccd9f8948f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:26:33.899871 20742 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8b0fa4edea40e29f540ce585be8b72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=867ebfbfbe1640e590ccabccd9f8948f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:26:33.899895 20882 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8b0fa4edea40e29f540ce585be8b72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=867ebfbfbe1640e590ccabccd9f8948f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:26:33.900031 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8b0fa4edea40e29f540ce585be8b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.900032 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8b0fa4edea40e29f540ce585be8b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.900049 20742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8b0fa4edea40e29f540ce585be8b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:33.901675 20975 tablet_bootstrap.cc:492] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: Bootstrap starting.
I20260924 19:26:33.901741 20972 tablet_bootstrap.cc:492] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: Bootstrap starting.
I20260924 19:26:33.901764 20971 tablet_bootstrap.cc:492] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: Bootstrap starting.
I20260924 19:26:33.902292 20975 tablet_bootstrap.cc:654] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.902607 20971 tablet_bootstrap.cc:654] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.902823 20972 tablet_bootstrap.cc:654] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:33.903324 20975 tablet_bootstrap.cc:492] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: No bootstrap required, opened a new log
I20260924 19:26:33.903395 20975 ts_tablet_manager.cc:1397] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:26:33.903553 20975 raft_consensus.cc:348] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.903633 20975 raft_consensus.cc:374] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.903666 20975 raft_consensus.cc:729] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b4232f5554028b9bfe4d261b2008b, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.903704 20975 consensus_queue.cc:260] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.903801 20975 ts_tablet_manager.cc:1428] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:33.903826 20971 tablet_bootstrap.cc:492] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: No bootstrap required, opened a new log
I20260924 19:26:33.903821 20981 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:33.903865 20971 ts_tablet_manager.cc:1397] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:33.903888 20981 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:33.904249 20971 raft_consensus.cc:348] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.904333 20971 raft_consensus.cc:374] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.904363 20971 raft_consensus.cc:729] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0db79d170da49a0a082957f9b691de0, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.904402 20971 consensus_queue.cc:260] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.904500 20971 ts_tablet_manager.cc:1428] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:26:33.904556 20977 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:33.904654 20977 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:33.904991 20972 tablet_bootstrap.cc:492] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: No bootstrap required, opened a new log
I20260924 19:26:33.905056 20972 ts_tablet_manager.cc:1397] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:26:33.905203 20972 raft_consensus.cc:348] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.905277 20972 raft_consensus.cc:374] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:33.905308 20972 raft_consensus.cc:729] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895e3b3b70e543c0ad38c24969dcdee2, State: Initialized, Role: FOLLOWER
I20260924 19:26:33.905364 20972 consensus_queue.cc:260] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:33.905450 20972 ts_tablet_manager.cc:1428] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:33.905573 20976 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:33.905687 20976 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:34.233870 20977 raft_consensus.cc:482] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:34.233968 20977 raft_consensus.cc:504] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:34.234120 20977 leader_election.cc:288] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667), 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329)
I20260924 19:26:34.234388 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8b0fa4edea40e29f540ce585be8b72" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895e3b3b70e543c0ad38c24969dcdee2" is_pre_election: true
I20260924 19:26:34.234447 20762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8b0fa4edea40e29f540ce585be8b72" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0b4232f5554028b9bfe4d261b2008b" is_pre_election: true
I20260924 19:26:34.234515 20622 raft_consensus.cc:2454] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 0.
I20260924 19:26:34.234548 20762 raft_consensus.cc:2454] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 0.
I20260924 19:26:34.234700 20836 leader_election.cc:302] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895e3b3b70e543c0ad38c24969dcdee2, e0db79d170da49a0a082957f9b691de0; no voters:
I20260924 19:26:34.234792 20977 raft_consensus.cc:2781] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:34.234835 20977 raft_consensus.cc:482] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:34.234849 20977 raft_consensus.cc:3037] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:34.235530 20977 raft_consensus.cc:504] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:34.235673 20977 leader_election.cc:288] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 election: Requested vote from peers 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667), 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329)
I20260924 19:26:34.235826 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8b0fa4edea40e29f540ce585be8b72" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895e3b3b70e543c0ad38c24969dcdee2"
I20260924 19:26:34.235872 20762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8b0fa4edea40e29f540ce585be8b72" candidate_uuid: "e0db79d170da49a0a082957f9b691de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0b4232f5554028b9bfe4d261b2008b"
I20260924 19:26:34.235899 20622 raft_consensus.cc:3037] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:34.235980 20762 raft_consensus.cc:3037] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:34.236586 20622 raft_consensus.cc:2454] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 1.
I20260924 19:26:34.236711 20762 raft_consensus.cc:2454] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0db79d170da49a0a082957f9b691de0 in term 1.
I20260924 19:26:34.236773 20836 leader_election.cc:302] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895e3b3b70e543c0ad38c24969dcdee2, e0db79d170da49a0a082957f9b691de0; no voters:
I20260924 19:26:34.236858 20977 raft_consensus.cc:2781] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:34.236912 20977 raft_consensus.cc:686] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [term 1 LEADER]: Becoming Leader. State: Replica: e0db79d170da49a0a082957f9b691de0, State: Running, Role: LEADER
I20260924 19:26:34.236959 20977 consensus_queue.cc:237] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } }
I20260924 19:26:34.237236 21002 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: TxnStatusTablet state changed. Reason: New leader e0db79d170da49a0a082957f9b691de0. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.237329 21002 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:26:34.237572 21015 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:26:34.237706 21015 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:26:34.237624 20465 catalog_manager.cc:5495] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0db79d170da49a0a082957f9b691de0 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } health_report { overall_health: HEALTHY } } }
I20260924 19:26:34.260074 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.257421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.18.229.131:38715)
0924 19:26:34.257559 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:34.257562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:34.257570 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:34.257665 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:34.257666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:34.257693 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:34.257774 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:34.257921 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.258440 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.259186 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.259399 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.259498 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.259694 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:34.259812 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:34.259890 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:34.259951 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:26:34.259973 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260924 19:26:34.260067 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.257409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:38715 (local address 127.0.0.1:34304)
0924 19:26:34.257489 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:34.257502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:34.257559 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:34.257776 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:34.257777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:34.257779 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:34.257835 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:34.257843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.258533 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.258535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:34.259088 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:34.259090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.259508 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.259509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:34.259572 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.259586 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:34.259887 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:34.259897 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:26:34.259949 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:26:34.259996 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:26:34.261521 20622 raft_consensus.cc:1261] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Refusing update from remote peer e0db79d170da49a0a082957f9b691de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:34.261610 20762 raft_consensus.cc:1261] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Refusing update from remote peer e0db79d170da49a0a082957f9b691de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:34.261785 20977 consensus_queue.cc:1035] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:34.261886 21002 consensus_queue.cc:1035] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:34.262481 20976 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: TxnStatusTablet state changed. Reason: New leader e0db79d170da49a0a082957f9b691de0. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.262595 20976 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:34.262565 20981 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: TxnStatusTablet state changed. Reason: New leader e0db79d170da49a0a082957f9b691de0. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.262648 20981 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:34.263422 21002 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.263502 21002 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:26:34.263790 20977 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.263872 20977 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P e0db79d170da49a0a082957f9b691de0: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:26:34.264307 20981 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.264407 20981 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P 3d0b4232f5554028b9bfe4d261b2008b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:34.264634 20976 tablet_replica.cc:439] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0db79d170da49a0a082957f9b691de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 } } peers { permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 } } peers { permanent_uuid: "e0db79d170da49a0a082957f9b691de0" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38715 } } }
I20260924 19:26:34.264717 20976 tablet_replica.cc:442] T 5d8b0fa4edea40e29f540ce585be8b72 P 895e3b3b70e543c0ad38c24969dcdee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:34.264323 21021 mvcc.cc:205] Tried to move back new op lower bound from 7332978664492863488 to 7332978664394977280. Current Snapshot: MvccSnapshot[applied={T|T < 7332978664492863488}]
I20260924 19:26:34.268302 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.265627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.18.229.131:38715)
0924 19:26:34.265665 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:34.265668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:34.265706 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:26:34.265783 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:34.265784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:34.265793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:34.265844 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:34.265963 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.266761 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.267473 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.267826 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.267896 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.268061 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:34.268095 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:34.268162 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:34.268215 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:26:34.268235 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:26:34.273456 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.270815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:33787 (local address 127.18.229.131:38715)
0924 19:26:34.270871 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:34.270873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:34.271021 (+ 148us) server_negotiation.cc:407] Connection header received
0924 19:26:34.271064 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:34.271065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:34.271073 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:34.271112 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:34.271294 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.271709 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.272815 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.272990 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.273041 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.273226 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:34.273247 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:34.273283 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:34.273362 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:26:34.273383 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260924 19:26:34.273558 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.270723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:38715 (local address 127.18.229.131:33787)
0924 19:26:34.270941 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:34.270951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:34.271013 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:34.271124 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:34.271125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:34.271127 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:34.271244 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:34.271245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.271774 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.271775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:34.272727 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:34.272730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.273059 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.273060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:34.273132 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.273147 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:34.273303 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:34.273304 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:26:34.273359 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:26:34.273410 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260924 19:26:34.275583 20762 raft_consensus.cc:1261] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Refusing update from remote peer e0db79d170da49a0a082957f9b691de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:34.275601 20622 raft_consensus.cc:1261] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Refusing update from remote peer e0db79d170da49a0a082957f9b691de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:34.275796 21002 consensus_queue.cc:1035] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0b4232f5554028b9bfe4d261b2008b" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:34.275874 20977 consensus_queue.cc:1035] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895e3b3b70e543c0ad38c24969dcdee2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:34.290208 20465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:34.290319 20465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:34.291018 20465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f2c4feb1a764e06aea1b6b3cbbd0754: Sending DeleteTablet for 3 replicas of tablet 79126dab226f4b2cb7c5e970cb6abfad
I20260924 19:26:34.291253 20602 tablet_service.cc:1514] Processing DeleteTablet for tablet 79126dab226f4b2cb7c5e970cb6abfad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:34 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:60164
I20260924 19:26:34.291280 20742 tablet_service.cc:1514] Processing DeleteTablet for tablet 79126dab226f4b2cb7c5e970cb6abfad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:34 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:60318
I20260924 19:26:34.291368 20882 tablet_service.cc:1514] Processing DeleteTablet for tablet 79126dab226f4b2cb7c5e970cb6abfad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:34 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:34296
I20260924 19:26:34.292023 21037 tablet_replica.cc:330] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: stopping tablet replica
I20260924 19:26:34.292089 21037 raft_consensus.cc:2229] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:34.292145 21037 raft_consensus.cc:2258] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:34.292243 21038 tablet_replica.cc:330] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: stopping tablet replica
I20260924 19:26:34.292299 21038 raft_consensus.cc:2229] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:34.292387 21038 raft_consensus.cc:2258] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:34.292642 21038 ts_tablet_manager.cc:1905] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:34.294289 21039 tablet_replica.cc:330] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: stopping tablet replica
I20260924 19:26:34.294378 21039 raft_consensus.cc:2229] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:34.294476 21039 raft_consensus.cc:2258] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:34.294649 21037 ts_tablet_manager.cc:1905] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:34.296241 21037 ts_tablet_manager.cc:1918] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:34.296303 21037 log.cc:1196] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/79126dab226f4b2cb7c5e970cb6abfad
I20260924 19:26:34.296530 21037 ts_tablet_manager.cc:1939] T 79126dab226f4b2cb7c5e970cb6abfad P 3d0b4232f5554028b9bfe4d261b2008b: Deleting consensus metadata
I20260924 19:26:34.296811 21038 ts_tablet_manager.cc:1918] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:34.296864 21038 log.cc:1196] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/79126dab226f4b2cb7c5e970cb6abfad
I20260924 19:26:34.296877 20452 catalog_manager.cc:4841] TS 3d0b4232f5554028b9bfe4d261b2008b (127.18.229.130:45329): tablet 79126dab226f4b2cb7c5e970cb6abfad (table test-table [id=f01a87d5ab1d46d1b978bcd2b90a79a2]) successfully deleted
I20260924 19:26:34.297068 21038 ts_tablet_manager.cc:1939] T 79126dab226f4b2cb7c5e970cb6abfad P 895e3b3b70e543c0ad38c24969dcdee2: Deleting consensus metadata
I20260924 19:26:34.297413 20452 catalog_manager.cc:4841] TS 895e3b3b70e543c0ad38c24969dcdee2 (127.18.229.129:34667): tablet 79126dab226f4b2cb7c5e970cb6abfad (table test-table [id=f01a87d5ab1d46d1b978bcd2b90a79a2]) successfully deleted
I20260924 19:26:34.297495 21039 ts_tablet_manager.cc:1905] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:26:34 dist-test-slave-r8cc krb5kdc[20420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:34.298977 21039 ts_tablet_manager.cc:1918] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:34.299034 21039 log.cc:1196] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/79126dab226f4b2cb7c5e970cb6abfad
I20260924 19:26:34.299259 21039 ts_tablet_manager.cc:1939] T 79126dab226f4b2cb7c5e970cb6abfad P e0db79d170da49a0a082957f9b691de0: Deleting consensus metadata
I20260924 19:26:34.299934 20452 catalog_manager.cc:4841] TS e0db79d170da49a0a082957f9b691de0 (127.18.229.131:38715): tablet 79126dab226f4b2cb7c5e970cb6abfad (table test-table [id=f01a87d5ab1d46d1b978bcd2b90a79a2]) successfully deleted
I20260924 19:26:34.302505 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.292295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.18.229.129:34667)
0924 19:26:34.292338 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:34.292340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:34.292390 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:26:34.292497 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:34.292498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:34.292512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:34.292564 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:34.293230 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.293763 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.294941 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.295245 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.295325 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.299132 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:34.299141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:34.299141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:34.299149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:34.301195 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:34.301628 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:34.301631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:34.301632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:34.301703 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:34.301930 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:34.301931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:34.301932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:34.302088 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:34.302266 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:34.302312 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 19:26:34.302398 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 19:26:34.302786 20662 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60166
I20260924 19:26:34.310534 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:34.303799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.18.229.190:42221)
0924 19:26:34.303860 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:34.303862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:34.303871 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:34.303939 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:34.303939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:34.303951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:34.304016 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:34.304589 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.305350 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.306092 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:34.306426 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:34.306496 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:34.307634 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:34.307636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:34.307636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:34.307643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:34.309446 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:34.309795 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:34.309796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:34.309797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:34.309856 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:34.310038 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:34.310038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:34.310039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:34.310166 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:34.310263 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:34.310347 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:26:34.310443 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 19:26:34.310837 20465 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41754
I20260924 19:26:34.311499 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 20537
I20260924 19:26:34.319888 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 20677
I20260924 19:26:34.325788 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 20817
I20260924 19:26:34.331212 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 20433
2026-09-24T19:26:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2851 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:34 dist-test-slave-r8cc krb5kdc[21046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:34 dist-test-slave-r8cc krb5kdc[21046](info): set up 2 sockets
Sep 24 19:26:34 dist-test-slave-r8cc krb5kdc[21046](info): commencing operation
krb5kdc: starting...
W20260924 19:26:35.371721 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:35.482314 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37855
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:37855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:35.561875 21059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:35.562073 21059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:35.562094 21059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:35.563158 21059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:35.563190 21059 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:26:35.563205 21059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:35.563216 21059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:35.563230 21059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:35.563241 21059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:35.564679 21059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:37855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:35.564860 21059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:35.565022 21059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:35.569736 21064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:35.569737 21067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:35.569830 21065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:35.569761 21059 server_base.cc:1034] running on GCE node
I20260924 19:26:35.570143 21059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:35.570454 21059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:35.571612 21059 hybrid_clock.cc:648] HybridClock initialized: now 1790277995571605 us; error 34 us; skew 500 ppm
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:35.574033 21059 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:35.574554 21059 webserver.cc:460] Webserver started at http://127.18.229.190:35757/ using document root <none> and password file <none>
I20260924 19:26:35.574774 21059 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:35.574839 21059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:35.574992 21059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:35.576056 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "cb35b858cfb5c43da3864d6566179f79"
server_key_iv: "54453b8b4512de511f2350b43727d137"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.576499 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "cb35b858cfb5c43da3864d6566179f79"
server_key_iv: "54453b8b4512de511f2350b43727d137"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.577770 21059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:35.578461 21074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:35.578624 21059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:35.578724 21059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "cb35b858cfb5c43da3864d6566179f79"
server_key_iv: "54453b8b4512de511f2350b43727d137"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.578794 21059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:35.595337 21059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:35.597302 21059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:35.597470 21059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:35.601989 21059 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:37855
I20260924 19:26:35.602054 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:37855 every 8 connection(s)
I20260924 19:26:35.602396 21059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:35.603003 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:35.605247 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Bootstrap starting.
I20260924 19:26:35.605868 21137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:35.606168 21137 log.cc:824] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:35.606698 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21059
I20260924 19:26:35.606765 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:35.606921 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: No bootstrap required, opened a new log
I20260924 19:26:35.606945 19350 external_mini_cluster.cc:1428] Setting key e11f9272e59fee1789ac674f4c3db553
I20260924 19:26:35.609903 21137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37855 } }
I20260924 19:26:35.610069 21137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:35.610102 21137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eaa0d0de5434f9ebdc9afe23b112fa7, State: Initialized, Role: FOLLOWER
I20260924 19:26:35.610224 21137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37855 } }
I20260924 19:26:35.610291 21137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:35.610317 21137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:35.610357 21137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:35.610996 21137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37855 } }
I20260924 19:26:35.611083 21137 leader_election.cc:302] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eaa0d0de5434f9ebdc9afe23b112fa7; no voters:
I20260924 19:26:35.611258 21137 leader_election.cc:288] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:35.611277 21142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:35.611377 21142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [term 1 LEADER]: Becoming Leader. State: Replica: 3eaa0d0de5434f9ebdc9afe23b112fa7, State: Running, Role: LEADER
I20260924 19:26:35.611464 21142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37855 } }
I20260924 19:26:35.611501 21137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:35.611758 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37855 } } }
I20260924 19:26:35.611963 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:35.611900 21144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eaa0d0de5434f9ebdc9afe23b112fa7. Latest consensus state: current_term: 1 leader_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eaa0d0de5434f9ebdc9afe23b112fa7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37855 } } }
I20260924 19:26:35.612142 21144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:35.612223 21149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:35.612696 21149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:35.614511 21149 catalog_manager.cc:1347] Generated new cluster ID: da9d3ab7aa484e25b5700259a8efbc48
I20260924 19:26:35.614562 21149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:35.618830 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.607450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.18.229.190:37855)
0924 19:26:35.607652 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:35.607662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:35.607694 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:26:35.608416 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:35.608426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:35.608489 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:35.608643 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:35.609240 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.610589 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.611462 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.612050 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.612150 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.615212 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:35.615216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:35.615217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:35.615228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:35.617168 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.617550 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.617553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.617553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.617661 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.617864 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.617865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.617865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.618173 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:35.618316 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:35.618443 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:26:35.618583 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260924 19:26:35.647624 21149 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:35.648365 21149 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:35.655738 21149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Generated new TSK 0
I20260924 19:26:35.655949 21149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:35.679030 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37855
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:35.762010 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:35.762213 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:35.762238 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:35.763424 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:35.763465 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:35.763485 21163 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:26:35.763522 21163 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:26:35.763557 21163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:35.764991 21163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:35.765172 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:35.765357 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:35.770337 21171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:35.770336 21169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:35.770390 21163 server_base.cc:1034] running on GCE node
W20260924 19:26:35.770344 21168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:35.770716 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:35.771001 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:35.772186 21163 hybrid_clock.cc:648] HybridClock initialized: now 1790277995772160 us; error 33 us; skew 500 ppm
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:35.774530 21163 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:35.775069 21163 webserver.cc:460] Webserver started at http://127.18.229.129:44693/ using document root <none> and password file <none>
I20260924 19:26:35.775311 21163 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:35.775378 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:35.775533 21163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:35.776643 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "060109aa68604959a544bac8060885d0"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "626b9dc2456452eb5dc133c5d8ebaeb3"
server_key_iv: "d08e794042ffc855aed51ceb86b064ef"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.777012 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "060109aa68604959a544bac8060885d0"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "626b9dc2456452eb5dc133c5d8ebaeb3"
server_key_iv: "d08e794042ffc855aed51ceb86b064ef"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.778275 21163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:35.778955 21178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:35.779121 21163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:35.779212 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "060109aa68604959a544bac8060885d0"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "626b9dc2456452eb5dc133c5d8ebaeb3"
server_key_iv: "d08e794042ffc855aed51ceb86b064ef"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.779292 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:35.801188 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:35.803153 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:35.803309 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:35.803797 21163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:35.803858 21163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:35.803928 21163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:35.803959 21163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:35.811161 21163 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:40547
I20260924 19:26:35.811256 21293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:40547 every 8 connection(s)
I20260924 19:26:35.811583 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:35.813977 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.804390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:58341 (local address 127.18.229.190:37855)
0924 19:26:35.804448 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:35.804450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:35.804646 (+ 196us) server_negotiation.cc:407] Connection header received
0924 19:26:35.805118 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:35.805120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:35.805135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:35.805210 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:35.806059 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.806926 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.807980 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.808278 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.808369 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.811143 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:35.811145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:35.811145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:35.811153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:35.812854 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.813218 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.813218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.813219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.813273 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.813482 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.813482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.813483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.813580 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:35.813630 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:35.813847 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 19:26:35.813914 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:26:35.814213 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.804362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.18.229.129:58341)
0924 19:26:35.804535 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:35.804567 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:35.805018 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:35.805315 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:35.805319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:35.805655 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:35.805949 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.805956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.807059 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.807060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.807857 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.807858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.808394 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.808395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.808522 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.809042 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:35.809056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:35.811045 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:35.812928 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.812934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.812936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.813149 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.813333 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.813334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.813334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.813429 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.813633 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:35.813637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:35.813744 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:26:35.813824 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:26:35.813951 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260924 19:26:35.814330 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21163
I20260924 19:26:35.814445 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:35.814637 19350 external_mini_cluster.cc:1428] Setting key 4841b7e86f4e78c177eb19eff2c18499
I20260924 19:26:35.815184 21294 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37855
I20260924 19:26:35.815258 21294 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:35.816980 21294 heartbeater.cc:511] Master 127.18.229.190:37855 requested a full tablet report, sending...
I20260924 19:26:35.817541 21091 ts_manager.cc:194] Registered new tserver with Master: 060109aa68604959a544bac8060885d0 (127.18.229.129:40547)
I20260924 19:26:35.818194 21091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:58341
I20260924 19:26:35.822940 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.816299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:58899 (local address 127.18.229.190:37855)
0924 19:26:35.816747 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:35.816749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:35.816756 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:26:35.816773 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:35.816774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:35.816781 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:35.816830 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:35.817230 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.817812 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.818947 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.819277 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.819327 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.820498 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:35.820499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:35.820499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:35.820505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:35.821852 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.822237 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.822238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.822238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.822283 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.822470 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.822471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.822471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.822558 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:35.822605 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:35.822804 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:26:35.822862 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430}
I20260924 19:26:35.823032 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.816233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.18.229.129:58899)
0924 19:26:35.816349 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:35.816359 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:35.816420 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:35.816856 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:35.816856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:35.817075 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:35.817153 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.817155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.817913 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.817914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.818834 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.818837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.819412 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.819413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.819501 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.819920 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:35.819926 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:35.820415 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:35.821937 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.821939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.821939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.822166 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.822349 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.822350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.822350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.822412 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.822620 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:35.822621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:35.822682 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 19:26:35.822779 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:26:35.822883 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:35.841192 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37855
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:35.922937 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:35.923166 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:35.923207 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:35.924417 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:35.924481 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:35.924520 21303 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:26:35.924551 21303 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:26:35.924584 21303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:35.926108 21303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:35.926374 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:35.926596 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:35.931703 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:35.931769 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:35.931931 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:35.931916 21303 server_base.cc:1034] running on GCE node
I20260924 19:26:35.932181 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:35.932446 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:35.933594 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790277995933578 us; error 32 us; skew 500 ppm
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:35.935994 21303 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:35.936508 21303 webserver.cc:460] Webserver started at http://127.18.229.130:42479/ using document root <none> and password file <none>
I20260924 19:26:35.936688 21303 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:35.936728 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:35.936815 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:35.937774 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "cd4fec8db3684ad987edfe541e02fa97"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "3b14380b2346c72482b1a910c80413b2"
server_key_iv: "a84c0474b0c018a7efb4a99246a6464b"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.938077 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "cd4fec8db3684ad987edfe541e02fa97"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "3b14380b2346c72482b1a910c80413b2"
server_key_iv: "a84c0474b0c018a7efb4a99246a6464b"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.939291 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:35.939916 21318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:35.940088 21303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:35.940227 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "cd4fec8db3684ad987edfe541e02fa97"
format_stamp: "Formatted at 2026-09-24 19:26:35 on dist-test-slave-r8cc"
server_key: "3b14380b2346c72482b1a910c80413b2"
server_key_iv: "a84c0474b0c018a7efb4a99246a6464b"
server_key_version: "encryptionkey@0"
I20260924 19:26:35.940313 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:35.958559 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:35.960328 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:35.960516 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:35.961016 21303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:35.961081 21303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:35.961127 21303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:35.961155 21303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:35.968397 21303 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:42247
I20260924 19:26:35.968477 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:42247 every 8 connection(s)
Sep 24 19:26:35 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:35.968806 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:35.971752 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.961569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:52395 (local address 127.18.229.190:37855)
0924 19:26:35.961614 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:35.961617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:35.961832 (+ 215us) server_negotiation.cc:407] Connection header received
0924 19:26:35.962337 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:35.962339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:35.962351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:35.962419 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:35.963342 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.964249 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.966063 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.966412 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.966478 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.969313 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:35.969315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:35.969315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:35.969323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:35.970406 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.970807 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.970808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.970809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.970868 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.971128 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.971129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.971130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.971243 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:35.971319 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:35.971583 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:26:35.971663 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:26:35.971939 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.961561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.18.229.130:52395)
0924 19:26:35.961712 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:35.961745 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:35.962234 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:35.962543 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:35.962547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:35.962939 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:35.963237 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.963245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.964339 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.964341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.965951 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.965954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.966502 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.966503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.966661 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.967292 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:35.967309 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:35.969234 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:35.970502 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.970509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.970511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.970728 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.970936 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.970937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.970937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.971054 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.971310 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:35.971316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:35.971455 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 19:26:35.971563 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:26:35.971712 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260924 19:26:35.972453 21434 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37855
I20260924 19:26:35.972532 21434 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:35.972707 21434 heartbeater.cc:511] Master 127.18.229.190:37855 requested a full tablet report, sending...
I20260924 19:26:35.973066 21091 ts_manager.cc:194] Registered new tserver with Master: cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247)
I20260924 19:26:35.973456 21091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:52395
I20260924 19:26:35.976760 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21303
I20260924 19:26:35.976869 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:35.977036 19350 external_mini_cluster.cc:1428] Setting key 113e1221096ced0ea89b833ae22e3998
I20260924 19:26:35.979007 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.973546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:43861 (local address 127.18.229.190:37855)
0924 19:26:35.973583 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:35.973585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:35.973667 (+ 82us) server_negotiation.cc:407] Connection header received
0924 19:26:35.973711 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:35.973712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:35.973721 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:35.973780 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:35.974091 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.974637 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.975326 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.975541 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.975598 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.976395 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:35.976396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:35.976396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:35.976401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:35.977878 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.978213 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.978214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.978214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.978265 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:35.978466 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:35.978467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:35.978467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:35.978610 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:35.978683 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:35.978844 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:26:35.978918 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:26:35.979015 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:35.973484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.18.229.130:43861)
0924 19:26:35.973592 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:35.973601 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:35.973660 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:35.973776 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:35.973777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:35.973968 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:35.974015 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.974016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.974719 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.974720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.975232 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:35.975233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:35.975608 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:35.975609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:35.975659 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:35.976005 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:35.976009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:35.976326 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:35.977963 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.977963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.977964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.978129 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.978337 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:35.978338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:35.978338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:35.978394 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:35.978689 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:35.978690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:35.978733 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:26:35.978818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:26:35.978926 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:36.000084 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37855
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:36.081099 21443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:36.081310 21443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:36.081334 21443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:36.082664 21443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:36.082708 21443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:36.082729 21443 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:26:36.082783 21443 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:26:36.082813 21443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:36.084379 21443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:36.084618 21443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:36.084857 21443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:36.089907 21448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:36.089888 21449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:36.089881 21451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:36.089946 21443 server_base.cc:1034] running on GCE node
I20260924 19:26:36.090190 21443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:36.090431 21443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:36.091588 21443 hybrid_clock.cc:648] HybridClock initialized: now 1790277996091581 us; error 42 us; skew 500 ppm
Sep 24 19:26:36 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277996, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:36.094292 21443 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:36.094749 21443 webserver.cc:460] Webserver started at http://127.18.229.131:44621/ using document root <none> and password file <none>
I20260924 19:26:36.094982 21443 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:36.095026 21443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:36.095175 21443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:36.096288 21443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "8a2b3856d81242ca8ab5116335cfc71c"
format_stamp: "Formatted at 2026-09-24 19:26:36 on dist-test-slave-r8cc"
server_key: "a3b5ba0935c0ae95ea23bf889cad940c"
server_key_iv: "339d5c085605b61fb24d0c4da61d51f0"
server_key_version: "encryptionkey@0"
I20260924 19:26:36.096642 21443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "8a2b3856d81242ca8ab5116335cfc71c"
format_stamp: "Formatted at 2026-09-24 19:26:36 on dist-test-slave-r8cc"
server_key: "a3b5ba0935c0ae95ea23bf889cad940c"
server_key_iv: "339d5c085605b61fb24d0c4da61d51f0"
server_key_version: "encryptionkey@0"
I20260924 19:26:36.097919 21443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:26:36.098683 21458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:36.098856 21443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:36.098937 21443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "8a2b3856d81242ca8ab5116335cfc71c"
format_stamp: "Formatted at 2026-09-24 19:26:36 on dist-test-slave-r8cc"
server_key: "a3b5ba0935c0ae95ea23bf889cad940c"
server_key_iv: "339d5c085605b61fb24d0c4da61d51f0"
server_key_version: "encryptionkey@0"
I20260924 19:26:36.098989 21443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:36.127885 21443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:36.129889 21443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:36.130036 21443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:36.130556 21443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:36.130596 21443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:36.130621 21443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:36.130635 21443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:36.137771 21443 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:41465
I20260924 19:26:36.137853 21573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:41465 every 8 connection(s)
Sep 24 19:26:36 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277996, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:36.138157 21443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:36.141119 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.131092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:53047 (local address 127.18.229.190:37855)
0924 19:26:36.131143 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.131146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.131396 (+ 250us) server_negotiation.cc:407] Connection header received
0924 19:26:36.131942 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.131943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.131957 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.132033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:36.133065 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.133922 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.135615 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.135929 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.135983 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.138706 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:36.138707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:36.138707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:36.138714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:36.139927 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.140335 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.140335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.140336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.140383 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.140603 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.140603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.140603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.140698 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:36.140742 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.140993 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.141056 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:26:36.141251 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.131102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.18.229.131:53047)
0924 19:26:36.131275 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.131308 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.131824 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.132178 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.132184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.132600 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:36.132930 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.132937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.134035 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.134036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.135506 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.135510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.136008 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.136010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.136205 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.136796 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:36.136806 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:36.138621 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:36.139998 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:36.140004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:36.140005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:36.140276 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:36.140466 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:36.140466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:36.140466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:36.140562 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:36.140747 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:36.140751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:36.140863 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.140946 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.141063 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260924 19:26:36.141736 21574 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37855
I20260924 19:26:36.141824 21574 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:36.142011 21574 heartbeater.cc:511] Master 127.18.229.190:37855 requested a full tablet report, sending...
I20260924 19:26:36.142377 21089 ts_manager.cc:194] Registered new tserver with Master: 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:36.142901 21089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:53047
I20260924 19:26:36.144740 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21443
I20260924 19:26:36.144837 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:36.144999 19350 external_mini_cluster.cc:1428] Setting key 899f90231fea84bfc00995a2b687be26
I20260924 19:26:36.145869 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:26:36.148496 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.142733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.18.229.131:39663)
0924 19:26:36.142847 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.142858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.142916 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.143057 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.143058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.143274 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:36.143358 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.143360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.144192 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.144193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.144682 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.144683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.145108 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.145108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.145205 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.145650 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:36.145654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:36.146020 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:36.147471 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:36.147472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:36.147473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:36.147706 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:36.147936 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:36.147937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:36.147937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:36.147985 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:36.148261 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:36.148262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:36.148297 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.148348 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.148423 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":23,"thread_start_us":62,"threads_started":1}
I20260924 19:26:36.148679 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.142801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:39663 (local address 127.18.229.190:37855)
0924 19:26:36.142838 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.142839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.142901 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:26:36.142987 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.142988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.142997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.143049 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:36.143431 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.144082 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.144768 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.145022 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.145107 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.146096 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:36.146097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:36.146097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:36.146103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:36.147394 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.147793 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.147794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.147795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.147859 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.148046 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.148047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.148048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.148197 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:36.148273 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.148378 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.148456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 19:26:36 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:26:36 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:36.168965 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.162361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.18.229.190:37855)
0924 19:26:36.162425 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.162426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.162436 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:36.162464 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.162465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.162474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.162526 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:36.163049 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.163531 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.164322 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.164504 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.164558 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.166640 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:36.166642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:36.166642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:36.166649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:36.168022 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.168358 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.168359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.168360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.168413 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.168556 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.168557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.168557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.168649 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:36.168717 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.168796 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.168851 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:26:36.170670 21089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:36.172223 21089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:36.178923 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.174697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40547 (local address 127.0.0.1:40902)
0924 19:26:36.174922 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.174931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.174983 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.175379 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.175381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.175384 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.175484 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.175486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.176849 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.176850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.177927 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.177929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.178581 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.178582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.178660 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.178673 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.178775 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.178828 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260924 19:26:36.179056 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.174817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.18.229.130:42247)
0924 19:26:36.175165 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.175169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.175177 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:36.175209 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.175210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.175223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.175372 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.175578 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.176467 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.177953 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.178687 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.178764 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.178816 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.178910 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.178940 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":83,"threads_started":1}
I20260924 19:26:36.179288 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.175209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.18.229.131:41465)
0924 19:26:36.175531 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.175571 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.175581 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:36.175620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.175622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.175637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.175712 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.175825 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.176893 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.177985 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.178685 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.178764 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.178816 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.179128 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.179179 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":80,"threads_started":1}
I20260924 19:26:36.179392 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.175005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.18.229.129:40547)
0924 19:26:36.175171 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.175176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.175187 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:26:36.175227 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.175230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.175247 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.175569 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.175634 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.176765 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.177999 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.178484 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.178582 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.178642 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.179222 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.179275 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260924 19:26:36.179062 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.174638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:42247 (local address 127.0.0.1:52978)
0924 19:26:36.174741 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.174781 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.174851 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.175375 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.175378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.175381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.175489 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.175491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.176549 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.176550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.177850 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.177853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.178772 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.178773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.178829 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.178831 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.178897 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.178931 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":44,"thread_start_us":59,"threads_started":1}
I20260924 19:26:36.179159 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.174742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.0.0.1:40440)
0924 19:26:36.175038 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.175044 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.175067 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.175727 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.175728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.175730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.175762 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.175763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.176982 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.176983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.177866 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.177868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.178960 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.178961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.179020 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.179025 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.179078 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.179108 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":77,"threads_started":1}
I20260924 19:26:36.180078 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 21c67368d55349e0834ad5110dc15081 (DEFAULT_TABLE table=test-table [id=8719ed75945e45a6b2a24139bb280008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:36.180220 21368 tablet_service.cc:1467] Processing CreateTablet for tablet 21c67368d55349e0834ad5110dc15081 (DEFAULT_TABLE table=test-table [id=8719ed75945e45a6b2a24139bb280008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:36.180505 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c67368d55349e0834ad5110dc15081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.180555 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c67368d55349e0834ad5110dc15081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.183583 21597 tablet_bootstrap.cc:492] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Bootstrap starting.
I20260924 19:26:36.183631 21598 tablet_bootstrap.cc:492] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Bootstrap starting.
I20260924 19:26:36.184309 21598 tablet_bootstrap.cc:654] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.184327 21597 tablet_bootstrap.cc:654] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.184453 21508 tablet_service.cc:1467] Processing CreateTablet for tablet 21c67368d55349e0834ad5110dc15081 (DEFAULT_TABLE table=test-table [id=8719ed75945e45a6b2a24139bb280008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:36.184621 21598 log.cc:824] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:36.184653 21597 log.cc:824] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:36.184803 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c67368d55349e0834ad5110dc15081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.185381 21598 tablet_bootstrap.cc:492] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: No bootstrap required, opened a new log
I20260924 19:26:36.185381 21597 tablet_bootstrap.cc:492] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: No bootstrap required, opened a new log
I20260924 19:26:36.185453 21597 ts_tablet_manager.cc:1397] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:36.185498 21598 ts_tablet_manager.cc:1397] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:36.187330 21601 tablet_bootstrap.cc:492] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Bootstrap starting.
I20260924 19:26:36.187515 21597 raft_consensus.cc:348] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.187688 21597 raft_consensus.cc:374] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.187606 21598 raft_consensus.cc:348] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.187726 21598 raft_consensus.cc:374] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.187724 21597 raft_consensus.cc:729] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060109aa68604959a544bac8060885d0, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.187763 21598 raft_consensus.cc:729] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.187847 21597 consensus_queue.cc:260] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.187884 21598 consensus_queue.cc:260] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.188130 21598 ts_tablet_manager.cc:1428] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:26:36.188130 21597 ts_tablet_manager.cc:1428] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:36.188199 21434 heartbeater.cc:503] Master 127.18.229.190:37855 was elected leader, sending a full tablet report...
I20260924 19:26:36.188256 21294 heartbeater.cc:503] Master 127.18.229.190:37855 was elected leader, sending a full tablet report...
I20260924 19:26:36.188696 21601 tablet_bootstrap.cc:654] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.189019 21601 log.cc:824] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:36.189817 21601 tablet_bootstrap.cc:492] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: No bootstrap required, opened a new log
I20260924 19:26:36.189868 21601 ts_tablet_manager.cc:1397] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:26:36.191341 21601 raft_consensus.cc:348] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.191473 21601 raft_consensus.cc:374] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.191496 21601 raft_consensus.cc:729] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.191603 21601 consensus_queue.cc:260] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.191809 21601 ts_tablet_manager.cc:1428] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:36.191887 21574 heartbeater.cc:503] Master 127.18.229.190:37855 was elected leader, sending a full tablet report...
I20260924 19:26:36.203114 21603 raft_consensus.cc:482] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:36.203209 21603 raft_consensus.cc:504] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.203464 21603 leader_election.cc:288] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:36.206363 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.203706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:49071 (local address 127.18.229.131:41465)
0924 19:26:36.203759 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.203761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.203776 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:26:36.203806 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.203806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.203817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.203888 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.204059 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.204663 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.205612 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.206017 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.206072 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.206079 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.206245 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.206284 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:26:36.206364 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.203657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.18.229.130:49071)
0924 19:26:36.203702 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.203713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.203755 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.203910 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.203911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.203913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.203979 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.203981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.204735 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.204736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.205528 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.205530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.206090 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.206091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.206173 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.206177 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.206229 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.206275 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:26:36.206480 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.203748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:38101 (local address 127.18.229.129:40547)
0924 19:26:36.203796 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.203798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.204081 (+ 283us) server_negotiation.cc:407] Connection header received
0924 19:26:36.204142 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.204143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.204154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.204212 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.204341 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.204840 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.205719 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.206102 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.206174 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.206181 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.206352 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.206406 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:26:36.206672 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.203658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40547 (local address 127.18.229.130:38101)
0924 19:26:36.203988 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.203997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.204067 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.204218 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.204218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.204220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.204281 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.204282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.204915 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.204915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.205638 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.205640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.206185 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.206185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.206275 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.206279 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.206337 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.206382 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":154,"thread_start_us":105,"threads_started":1}
I20260924 19:26:36.206729 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c67368d55349e0834ad5110dc15081" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2b3856d81242ca8ab5116335cfc71c" is_pre_election: true
I20260924 19:26:36.206892 21528 raft_consensus.cc:2454] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 0.
I20260924 19:26:36.206979 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c67368d55349e0834ad5110dc15081" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0" is_pre_election: true
I20260924 19:26:36.207141 21248 raft_consensus.cc:2454] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 0.
I20260924 19:26:36.207114 21319 leader_election.cc:302] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2b3856d81242ca8ab5116335cfc71c, cd4fec8db3684ad987edfe541e02fa97; no voters:
I20260924 19:26:36.207267 21603 raft_consensus.cc:2781] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:36.207329 21603 raft_consensus.cc:482] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:36.207365 21603 raft_consensus.cc:3037] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.208046 21603 raft_consensus.cc:504] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.208215 21603 leader_election.cc:288] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:36.208329 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c67368d55349e0834ad5110dc15081" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0"
I20260924 19:26:36.208379 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c67368d55349e0834ad5110dc15081" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2b3856d81242ca8ab5116335cfc71c"
I20260924 19:26:36.208415 21248 raft_consensus.cc:3037] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.208459 21528 raft_consensus.cc:3037] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.209082 21248 raft_consensus.cc:2454] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 1.
I20260924 19:26:36.209151 21528 raft_consensus.cc:2454] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 1.
I20260924 19:26:36.209647 21322 leader_election.cc:302] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, cd4fec8db3684ad987edfe541e02fa97; no voters:
I20260924 19:26:36.209884 21603 raft_consensus.cc:2781] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:36.210041 21603 raft_consensus.cc:686] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 1 LEADER]: Becoming Leader. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Running, Role: LEADER
I20260924 19:26:36.210134 21603 consensus_queue.cc:237] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.210992 21091 catalog_manager.cc:5495] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "cd4fec8db3684ad987edfe541e02fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } health_report { overall_health: UNKNOWN } } }
W20260924 19:26:36.219411 21435 tablet.cc:2369] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:26:36 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277995, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:36.255587 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.249047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.18.229.190:37855)
0924 19:26:36.249116 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.249118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.249146 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:26:36.249246 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.249247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.249258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.249333 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:36.249808 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.250293 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.250995 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.251184 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.251275 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.253247 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:36.253248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:36.253249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:36.253256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:36.254399 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.254774 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.254775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.254775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.254830 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:36.255039 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:36.255040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:36.255041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:36.255155 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:36.255228 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.255389 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.255474 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:26:36.255672 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.248835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37855 (local address 127.0.0.1:52276)
0924 19:26:36.249051 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.249082 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.249167 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.249340 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.249341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.249672 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:36.249717 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.249720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.250386 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.250387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.250910 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.250911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.251289 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.251290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.251349 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.251814 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:36.251818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:36.253154 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:36.254493 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:36.254500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:36.254503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:36.254687 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:36.254912 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:36.254913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:36.254914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:36.254977 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:36.255233 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:36.255240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:36.255283 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.255366 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.255550 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":79,"thread_start_us":92,"threads_started":1}
I20260924 19:26:36.257419 21091 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:52276:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:26:36.257601 21091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:36.259064 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 43a777b1cc6b4f30a86dac4f22a99338 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bee006ce79bf4d84871c8b538842c096]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:26:36.259088 21508 tablet_service.cc:1467] Processing CreateTablet for tablet 43a777b1cc6b4f30a86dac4f22a99338 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bee006ce79bf4d84871c8b538842c096]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:26:36.259151 21368 tablet_service.cc:1467] Processing CreateTablet for tablet 43a777b1cc6b4f30a86dac4f22a99338 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bee006ce79bf4d84871c8b538842c096]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:26:36.259235 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a777b1cc6b4f30a86dac4f22a99338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.259246 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a777b1cc6b4f30a86dac4f22a99338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.259284 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a777b1cc6b4f30a86dac4f22a99338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.260521 21601 tablet_bootstrap.cc:492] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: Bootstrap starting.
I20260924 19:26:36.260787 21597 tablet_bootstrap.cc:492] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: Bootstrap starting.
I20260924 19:26:36.260818 21598 tablet_bootstrap.cc:492] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: Bootstrap starting.
I20260924 19:26:36.261173 21601 tablet_bootstrap.cc:654] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.261216 21598 tablet_bootstrap.cc:654] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.261509 21597 tablet_bootstrap.cc:654] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.261794 21598 tablet_bootstrap.cc:492] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: No bootstrap required, opened a new log
I20260924 19:26:36.261859 21598 ts_tablet_manager.cc:1397] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:36.262039 21598 raft_consensus.cc:348] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.262120 21598 raft_consensus.cc:374] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.262142 21598 raft_consensus.cc:729] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.262179 21597 tablet_bootstrap.cc:492] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: No bootstrap required, opened a new log
I20260924 19:26:36.262185 21598 consensus_queue.cc:260] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.262234 21597 ts_tablet_manager.cc:1397] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:36.262272 21598 ts_tablet_manager.cc:1428] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:36.262362 21603 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.262399 21597 raft_consensus.cc:348] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.262463 21597 raft_consensus.cc:374] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.262456 21603 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.262485 21597 raft_consensus.cc:729] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060109aa68604959a544bac8060885d0, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.262532 21597 consensus_queue.cc:260] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.262615 21601 tablet_bootstrap.cc:492] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: No bootstrap required, opened a new log
I20260924 19:26:36.262630 21597 ts_tablet_manager.cc:1428] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:36.262661 21601 ts_tablet_manager.cc:1397] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:36.262689 21602 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.262766 21602 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.262809 21601 raft_consensus.cc:348] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.262885 21601 raft_consensus.cc:374] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.262907 21601 raft_consensus.cc:729] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.262948 21601 consensus_queue.cc:260] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.263046 21601 ts_tablet_manager.cc:1428] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:36.263114 21605 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.263206 21605 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.297237 21605 raft_consensus.cc:482] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:36.297346 21605 raft_consensus.cc:504] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.297703 21605 leader_election.cc:288] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247)
I20260924 19:26:36.300830 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.297994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:52361 (local address 127.18.229.129:40547)
0924 19:26:36.298036 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.298038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.298178 (+ 140us) server_negotiation.cc:407] Connection header received
0924 19:26:36.298210 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.298211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.298222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.298303 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.298476 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.299041 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.300005 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.300477 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.300559 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.300592 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.300706 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.300753 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:26:36.300848 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.297885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40547 (local address 127.18.229.131:52361)
0924 19:26:36.298049 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.298059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.298118 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.298322 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.298323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.298324 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.298404 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.298405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.299133 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.299134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.299876 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.299878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.300579 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.300580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.300630 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.300633 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.300677 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.300707 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260924 19:26:36.300895 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.297951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:56753 (local address 127.18.229.130:42247)
0924 19:26:36.297999 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.298001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.298033 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:26:36.298063 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.298064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.298075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.298127 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:36.298343 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.298864 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.299976 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.300425 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.300482 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.300489 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.300778 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.300823 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:36.301072 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a777b1cc6b4f30a86dac4f22a99338" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0" is_pre_election: true
I20260924 19:26:36.301136 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.297866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:42247 (local address 127.18.229.131:56753)
0924 19:26:36.297928 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.297942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.297994 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.298172 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.298174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.298176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:36.298263 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.298265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.298939 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.298940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.299870 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.299872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.300502 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.300503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.300649 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.300652 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.300737 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.300790 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":22}
I20260924 19:26:36.301165 21248 raft_consensus.cc:2454] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 0.
I20260924 19:26:36.301354 21462 leader_election.cc:302] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, 8a2b3856d81242ca8ab5116335cfc71c; no voters:
I20260924 19:26:36.301498 21605 raft_consensus.cc:2781] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:36.301560 21605 raft_consensus.cc:482] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:36.301592 21605 raft_consensus.cc:3037] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.301548 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a777b1cc6b4f30a86dac4f22a99338" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4fec8db3684ad987edfe541e02fa97" is_pre_election: true
I20260924 19:26:36.301703 21388 raft_consensus.cc:2454] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 0.
I20260924 19:26:36.302394 21605 raft_consensus.cc:504] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.302573 21605 leader_election.cc:288] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 election: Requested vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247)
I20260924 19:26:36.302721 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a777b1cc6b4f30a86dac4f22a99338" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0"
I20260924 19:26:36.302731 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a777b1cc6b4f30a86dac4f22a99338" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4fec8db3684ad987edfe541e02fa97"
I20260924 19:26:36.302810 21388 raft_consensus.cc:3037] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.302810 21248 raft_consensus.cc:3037] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.303637 21248 raft_consensus.cc:2454] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 1.
I20260924 19:26:36.303702 21388 raft_consensus.cc:2454] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 1.
I20260924 19:26:36.303828 21462 leader_election.cc:302] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, 8a2b3856d81242ca8ab5116335cfc71c; no voters:
I20260924 19:26:36.303979 21605 raft_consensus.cc:2781] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:36.304220 21605 raft_consensus.cc:686] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 LEADER]: Becoming Leader. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Running, Role: LEADER
I20260924 19:26:36.304361 21605 consensus_queue.cc:237] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.304836 21619 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: TxnStatusTablet state changed. Reason: New leader 8a2b3856d81242ca8ab5116335cfc71c. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.304930 21619 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:26:36.305116 21621 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:26:36.305203 21091 catalog_manager.cc:5495] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131). New cstate: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } health_report { overall_health: HEALTHY } } }
I20260924 19:26:36.305295 21621 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260924 19:26:36.312266 21295 tablet.cc:2369] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:36.322281 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.319646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.0.0.1:40446)
0924 19:26:36.319721 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.319733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.319781 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.319975 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.319976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.319978 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:36.320025 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.320027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.320723 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.320724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.321344 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.321346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.321739 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.321746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.321818 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.321849 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.322105 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.322114 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.322161 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.322208 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:26:36.322278 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.319733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.18.229.131:41465)
0924 19:26:36.319779 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.319781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.319790 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:36.319862 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.319863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.319882 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.319954 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:36.320098 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.320646 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.321436 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.321660 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.321727 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.321942 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.322041 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.322126 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.322169 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.322188 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:36.324313 21248 raft_consensus.cc:1261] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Refusing update from remote peer 8a2b3856d81242ca8ab5116335cfc71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:36.324605 21388 raft_consensus.cc:1261] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Refusing update from remote peer 8a2b3856d81242ca8ab5116335cfc71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:36.324612 21619 consensus_queue.cc:1035] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:36.324826 21619 consensus_queue.cc:1035] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:36.325711 21602 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: TxnStatusTablet state changed. Reason: New leader 8a2b3856d81242ca8ab5116335cfc71c. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.325800 21602 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.326705 21619 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.326797 21619 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:26:36.326817 21603 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: TxnStatusTablet state changed. Reason: New leader 8a2b3856d81242ca8ab5116335cfc71c. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.326903 21603 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.327322 21602 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.327404 21625 mvcc.cc:205] Tried to move back new op lower bound from 7332978672941490176 to 7332978672863494144. Current Snapshot: MvccSnapshot[applied={T|T < 7332978672941490176}]
I20260924 19:26:36.327412 21602 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 060109aa68604959a544bac8060885d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.328629 21605 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.328719 21605 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P 8a2b3856d81242ca8ab5116335cfc71c: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:26:36.329227 21603 tablet_replica.cc:439] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } } }
I20260924 19:26:36.329330 21603 tablet_replica.cc:442] T 43a777b1cc6b4f30a86dac4f22a99338 P cd4fec8db3684ad987edfe541e02fa97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:26:36.329587 21627 mvcc.cc:205] Tried to move back new op lower bound from 7332978672941490176 to 7332978672863494144. Current Snapshot: MvccSnapshot[applied={T|T < 7332978672941490176}]
I20260924 19:26:36.332327 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.329887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.18.229.130:42247)
0924 19:26:36.330062 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.330063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.330069 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:26:36.330090 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.330090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.330103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.330180 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:36.330283 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.330745 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.331524 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.331733 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.331820 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.331978 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.332080 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.332211 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.332235 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.332255 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 19:26:36.337020 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.334714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:47901 (local address 127.18.229.131:41465)
0924 19:26:36.334759 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.334762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.334770 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:36.334828 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.334829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.334839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.334906 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:36.335058 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.335511 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.336312 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.336507 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.336591 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.336775 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.336813 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.336875 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.336933 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.336950 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:36.337030 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.334642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.18.229.130:47901)
0924 19:26:36.334716 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.334726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.334768 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.334912 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.334913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.334914 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:36.334967 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.334969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.335591 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.335592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.336222 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.336224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.336593 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.336594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.336657 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.336677 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.336875 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.336877 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.336929 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.336968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:26:36.339409 21248 raft_consensus.cc:1261] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Refusing update from remote peer cd4fec8db3684ad987edfe541e02fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:36.339530 21528 raft_consensus.cc:1261] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Refusing update from remote peer cd4fec8db3684ad987edfe541e02fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:36.339675 21607 consensus_queue.cc:1035] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:36.339777 21603 consensus_queue.cc:1035] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:36.341733 21624 mvcc.cc:205] Tried to move back new op lower bound from 7332978673004625920 to 7332978672477523968. Current Snapshot: MvccSnapshot[applied={T|T < 7332978673004625920}]
I20260924 19:26:36.348150 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.345703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:42247 (local address 127.18.229.131:57795)
0924 19:26:36.345773 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.345785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.345824 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.346007 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.346008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.346009 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:36.346064 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.346065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.346667 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.346667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.347225 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.347226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.347698 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.347698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.347779 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.347794 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.347983 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.347985 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.348021 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.348062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:26:36.348165 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.345746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:57795 (local address 127.18.229.130:42247)
0924 19:26:36.345902 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.345905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.345917 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:26:36.345933 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.345934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.345941 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.346004 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:36.346123 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.346578 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.347302 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.347548 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.347628 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.347877 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.347918 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.347967 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.348033 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.348056 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260924 19:26:36.358250 21089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:36.358366 21089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:36.359086 21089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Sending DeleteTablet for 3 replicas of tablet 21c67368d55349e0834ad5110dc15081
I20260924 19:26:36.359431 21508 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c67368d55349e0834ad5110dc15081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:36 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40440
I20260924 19:26:36.359644 21642 tablet_replica.cc:330] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: stopping tablet replica
I20260924 19:26:36.359743 21642 raft_consensus.cc:2229] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:36.359799 21642 raft_consensus.cc:2258] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:36.360018 21228 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c67368d55349e0834ad5110dc15081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:36 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40902
I20260924 19:26:36.360365 21642 ts_tablet_manager.cc:1905] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:36.360680 21368 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c67368d55349e0834ad5110dc15081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:36 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:52978
I20260924 19:26:36.360998 21089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:36.361145 21089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:36.362236 21642 ts_tablet_manager.cc:1918] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:36.362308 21642 log.cc:1196] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/21c67368d55349e0834ad5110dc15081
I20260924 19:26:36.362564 21642 ts_tablet_manager.cc:1939] T 21c67368d55349e0834ad5110dc15081 P 8a2b3856d81242ca8ab5116335cfc71c: Deleting consensus metadata
I20260924 19:26:36.362547 21643 tablet_replica.cc:330] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: stopping tablet replica
I20260924 19:26:36.362775 21643 raft_consensus.cc:2229] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:36.362916 21643 raft_consensus.cc:2258] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:36.362947 21508 tablet_service.cc:1467] Processing CreateTablet for tablet fee9397bf4034187ab3c3a867ac45bff (DEFAULT_TABLE table=test-table [id=fa8cee91367c4e639e26882b449ca347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:36.363022 21228 tablet_service.cc:1467] Processing CreateTablet for tablet fee9397bf4034187ab3c3a867ac45bff (DEFAULT_TABLE table=test-table [id=fa8cee91367c4e639e26882b449ca347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:36.363059 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee9397bf4034187ab3c3a867ac45bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.363148 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee9397bf4034187ab3c3a867ac45bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.364238 21368 tablet_service.cc:1467] Processing CreateTablet for tablet fee9397bf4034187ab3c3a867ac45bff (DEFAULT_TABLE table=test-table [id=fa8cee91367c4e639e26882b449ca347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:36.364300 21601 tablet_bootstrap.cc:492] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Bootstrap starting.
I20260924 19:26:36.364389 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee9397bf4034187ab3c3a867ac45bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:36.364545 21597 tablet_bootstrap.cc:492] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Bootstrap starting.
I20260924 19:26:36.364677 21601 tablet_bootstrap.cc:654] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.364964 21597 tablet_bootstrap.cc:654] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.365180 21643 ts_tablet_manager.cc:1905] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:36.365583 21075 catalog_manager.cc:4841] TS 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465): tablet 21c67368d55349e0834ad5110dc15081 (table test-table [id=8719ed75945e45a6b2a24139bb280008]) successfully deleted
I20260924 19:26:36.365962 21598 tablet_bootstrap.cc:492] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Bootstrap starting.
I20260924 19:26:36.366411 21598 tablet_bootstrap.cc:654] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:36.366775 21643 ts_tablet_manager.cc:1918] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:36.366852 21643 log.cc:1196] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/21c67368d55349e0834ad5110dc15081
I20260924 19:26:36.367133 21643 ts_tablet_manager.cc:1939] T 21c67368d55349e0834ad5110dc15081 P 060109aa68604959a544bac8060885d0: Deleting consensus metadata
I20260924 19:26:36.367487 21598 tablet_bootstrap.cc:492] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: No bootstrap required, opened a new log
I20260924 19:26:36.367547 21598 ts_tablet_manager.cc:1397] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:36.367532 21078 catalog_manager.cc:4841] TS 060109aa68604959a544bac8060885d0 (127.18.229.129:40547): tablet 21c67368d55349e0834ad5110dc15081 (table test-table [id=8719ed75945e45a6b2a24139bb280008]) successfully deleted
I20260924 19:26:36.367712 21598 raft_consensus.cc:348] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.367769 21601 tablet_bootstrap.cc:492] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: No bootstrap required, opened a new log
I20260924 19:26:36.367802 21598 raft_consensus.cc:374] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.367810 21601 ts_tablet_manager.cc:1397] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:26:36.367835 21598 raft_consensus.cc:729] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.367879 21598 consensus_queue.cc:260] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.367949 21601 raft_consensus.cc:348] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.367985 21598 ts_tablet_manager.cc:1428] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:36.368026 21601 raft_consensus.cc:374] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.368054 21601 raft_consensus.cc:729] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.368094 21601 consensus_queue.cc:260] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.368207 21601 ts_tablet_manager.cc:1428] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:36.369063 21597 tablet_bootstrap.cc:492] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: No bootstrap required, opened a new log
I20260924 19:26:36.369140 21597 ts_tablet_manager.cc:1397] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 19:26:36.369292 21597 raft_consensus.cc:348] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.369369 21597 raft_consensus.cc:374] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:36.369397 21597 raft_consensus.cc:729] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060109aa68604959a544bac8060885d0, State: Initialized, Role: FOLLOWER
I20260924 19:26:36.369448 21597 consensus_queue.cc:260] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.369561 21597 ts_tablet_manager.cc:1428] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:36.372216 21644 tablet_replica.cc:330] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: stopping tablet replica
I20260924 19:26:36.372331 21644 raft_consensus.cc:2229] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:36.372459 21644 raft_consensus.cc:2258] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:36.372942 21644 ts_tablet_manager.cc:1905] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:36.374701 21644 ts_tablet_manager.cc:1918] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:36.374774 21644 log.cc:1196] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/21c67368d55349e0834ad5110dc15081
I20260924 19:26:36.375026 21644 ts_tablet_manager.cc:1939] T 21c67368d55349e0834ad5110dc15081 P cd4fec8db3684ad987edfe541e02fa97: Deleting consensus metadata
I20260924 19:26:36.375423 21077 catalog_manager.cc:4841] TS cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247): tablet 21c67368d55349e0834ad5110dc15081 (table test-table [id=8719ed75945e45a6b2a24139bb280008]) successfully deleted
W20260924 19:26:36.388718 21575 tablet.cc:2369] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:36.511422 21631 raft_consensus.cc:482] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:36.511516 21631 raft_consensus.cc:504] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.511718 21631 leader_election.cc:288] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247)
I20260924 19:26:36.511962 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee9397bf4034187ab3c3a867ac45bff" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4fec8db3684ad987edfe541e02fa97" is_pre_election: true
I20260924 19:26:36.511986 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee9397bf4034187ab3c3a867ac45bff" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0" is_pre_election: true
I20260924 19:26:36.512064 21388 raft_consensus.cc:2454] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 0.
I20260924 19:26:36.512094 21248 raft_consensus.cc:2454] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 0.
I20260924 19:26:36.512267 21462 leader_election.cc:302] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, 8a2b3856d81242ca8ab5116335cfc71c; no voters:
I20260924 19:26:36.512388 21631 raft_consensus.cc:2781] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:36.512427 21631 raft_consensus.cc:482] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:36.512447 21631 raft_consensus.cc:3037] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.513207 21631 raft_consensus.cc:504] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.513396 21631 leader_election.cc:288] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 election: Requested vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247)
I20260924 19:26:36.513545 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee9397bf4034187ab3c3a867ac45bff" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0"
I20260924 19:26:36.513626 21248 raft_consensus.cc:3037] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.513566 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee9397bf4034187ab3c3a867ac45bff" candidate_uuid: "8a2b3856d81242ca8ab5116335cfc71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4fec8db3684ad987edfe541e02fa97"
I20260924 19:26:36.513679 21388 raft_consensus.cc:3037] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:36.514251 21248 raft_consensus.cc:2454] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 1.
I20260924 19:26:36.514353 21388 raft_consensus.cc:2454] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2b3856d81242ca8ab5116335cfc71c in term 1.
I20260924 19:26:36.514432 21462 leader_election.cc:302] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, 8a2b3856d81242ca8ab5116335cfc71c; no voters:
I20260924 19:26:36.514549 21631 raft_consensus.cc:2781] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:36.514614 21631 raft_consensus.cc:686] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 1 LEADER]: Becoming Leader. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Running, Role: LEADER
I20260924 19:26:36.514667 21631 consensus_queue.cc:237] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:36.515247 21089 catalog_manager.cc:5495] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131). New cstate: current_term: 1 leader_uuid: "8a2b3856d81242ca8ab5116335cfc71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } health_report { overall_health: HEALTHY } } }
I20260924 19:26:36.551157 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.548886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.18.229.131:41465)
0924 19:26:36.548929 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.548932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.548940 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:36.549003 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.549004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.549015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.549068 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:36.549172 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.549662 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.550421 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.550630 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.550715 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.550866 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.550907 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.550976 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.551036 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.551054 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:26:36.554962 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.553224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:36335 (local address 127.18.229.131:41465)
0924 19:26:36.553254 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:36.553256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:36.553264 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:36.553334 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:36.553335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:36.553341 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:36.553373 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:36.553456 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.553864 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.554494 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.554656 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.554702 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.554801 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.554820 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.554847 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:36.554884 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 19:26:36.554903 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:26:36.554971 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:36.553162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.18.229.131:36335)
0924 19:26:36.553220 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:36.553231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:36.553272 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:36.553389 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:36.553390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:36.553391 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:36.553423 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.553424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.553922 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.553922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.554430 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:36.554432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:36.554716 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:36.554717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:36.554750 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:36.554754 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:36.554853 (+ 99us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:36.554854 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:26:36.554885 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 19:26:36.554911 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:26:36.556716 21248 raft_consensus.cc:1261] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Refusing update from remote peer 8a2b3856d81242ca8ab5116335cfc71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:36.556716 21388 raft_consensus.cc:1261] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Refusing update from remote peer 8a2b3856d81242ca8ab5116335cfc71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:36.556952 21619 consensus_queue.cc:1035] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:36.557078 21605 consensus_queue.cc:1035] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:26:36.561367 21624 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:26:38.081845 21091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:38.081971 21091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:38.082729 21091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Sending DeleteTablet for 3 replicas of tablet fee9397bf4034187ab3c3a867ac45bff
I20260924 19:26:38.083006 21368 tablet_service.cc:1514] Processing DeleteTablet for tablet fee9397bf4034187ab3c3a867ac45bff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:38 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:52978
I20260924 19:26:38.083050 21228 tablet_service.cc:1514] Processing DeleteTablet for tablet fee9397bf4034187ab3c3a867ac45bff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:38 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40902
I20260924 19:26:38.083232 21672 tablet_replica.cc:330] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: stopping tablet replica
I20260924 19:26:38.083240 21508 tablet_service.cc:1514] Processing DeleteTablet for tablet fee9397bf4034187ab3c3a867ac45bff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:38 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40440
I20260924 19:26:38.083261 21673 tablet_replica.cc:330] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: stopping tablet replica
I20260924 19:26:38.083335 21672 raft_consensus.cc:2229] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:38.083384 21672 raft_consensus.cc:2258] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:38.083425 21673 raft_consensus.cc:2229] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:38.083540 21673 raft_consensus.cc:2258] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:38.083652 21672 ts_tablet_manager.cc:1905] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:38.083846 21673 ts_tablet_manager.cc:1905] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:38.085493 21672 ts_tablet_manager.cc:1918] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:26:38.085570 21672 log.cc:1196] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/fee9397bf4034187ab3c3a867ac45bff
I20260924 19:26:38.085785 21673 ts_tablet_manager.cc:1918] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:26:38.085824 21672 ts_tablet_manager.cc:1939] T fee9397bf4034187ab3c3a867ac45bff P cd4fec8db3684ad987edfe541e02fa97: Deleting consensus metadata
I20260924 19:26:38.085868 21673 log.cc:1196] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/fee9397bf4034187ab3c3a867ac45bff
I20260924 19:26:38.086102 21673 ts_tablet_manager.cc:1939] T fee9397bf4034187ab3c3a867ac45bff P 060109aa68604959a544bac8060885d0: Deleting consensus metadata
I20260924 19:26:38.086145 21077 catalog_manager.cc:4841] TS cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247): tablet fee9397bf4034187ab3c3a867ac45bff (table test-table [id=fa8cee91367c4e639e26882b449ca347]) successfully deleted
I20260924 19:26:38.086530 21078 catalog_manager.cc:4841] TS 060109aa68604959a544bac8060885d0 (127.18.229.129:40547): tablet fee9397bf4034187ab3c3a867ac45bff (table test-table [id=fa8cee91367c4e639e26882b449ca347]) successfully deleted
I20260924 19:26:38.087111 21674 tablet_replica.cc:330] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: stopping tablet replica
I20260924 19:26:38.087164 21674 raft_consensus.cc:2229] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:38.087271 21674 raft_consensus.cc:2258] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:38.087502 21674 ts_tablet_manager.cc:1905] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:38.088852 21674 ts_tablet_manager.cc:1918] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:26:38.088904 21674 log.cc:1196] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/fee9397bf4034187ab3c3a867ac45bff
I20260924 19:26:38.089160 21674 ts_tablet_manager.cc:1939] T fee9397bf4034187ab3c3a867ac45bff P 8a2b3856d81242ca8ab5116335cfc71c: Deleting consensus metadata
I20260924 19:26:38.089484 21075 catalog_manager.cc:4841] TS 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465): tablet fee9397bf4034187ab3c3a867ac45bff (table test-table [id=fa8cee91367c4e639e26882b449ca347]) successfully deleted
Sep 24 19:26:38 dist-test-slave-r8cc krb5kdc[21046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:26:38 dist-test-slave-r8cc krb5kdc[21046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:38.112403 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.100902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.18.229.190:37855)
0924 19:26:38.101048 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.101050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.101059 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:38.101127 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.101128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.101138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.101242 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:38.101772 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.102276 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.103054 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.103255 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.103324 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.110019 (+ 6695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:38.110020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:38.110021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:38.110028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:38.111274 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:38.111705 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:38.111706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:38.111706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:38.111775 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:38.111959 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:38.111960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:38.111960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:38.112059 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:38.112222 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.112270 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.112333 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":92,"threads_started":1}
I20260924 19:26:38.113782 21091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:38.113955 21091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:38.115262 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 22c17b91692c4f3a8a6be334bd48a69e (DEFAULT_TABLE table=test-table [id=87bda96dd8164f7db71a17b78afef0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:38.115298 21368 tablet_service.cc:1467] Processing CreateTablet for tablet 22c17b91692c4f3a8a6be334bd48a69e (DEFAULT_TABLE table=test-table [id=87bda96dd8164f7db71a17b78afef0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:38.115365 21508 tablet_service.cc:1467] Processing CreateTablet for tablet 22c17b91692c4f3a8a6be334bd48a69e (DEFAULT_TABLE table=test-table [id=87bda96dd8164f7db71a17b78afef0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:38.115397 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22c17b91692c4f3a8a6be334bd48a69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:38.115439 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22c17b91692c4f3a8a6be334bd48a69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:38.115463 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22c17b91692c4f3a8a6be334bd48a69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:38.116750 21683 tablet_bootstrap.cc:492] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Bootstrap starting.
I20260924 19:26:38.116907 21684 tablet_bootstrap.cc:492] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Bootstrap starting.
I20260924 19:26:38.117172 21683 tablet_bootstrap.cc:654] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:38.117301 21684 tablet_bootstrap.cc:654] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:38.117616 21685 tablet_bootstrap.cc:492] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Bootstrap starting.
I20260924 19:26:38.117756 21683 tablet_bootstrap.cc:492] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: No bootstrap required, opened a new log
I20260924 19:26:38.117815 21683 ts_tablet_manager.cc:1397] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:38.117991 21685 tablet_bootstrap.cc:654] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:38.117972 21683 raft_consensus.cc:348] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.118039 21683 raft_consensus.cc:374] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:38.118063 21683 raft_consensus.cc:729] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060109aa68604959a544bac8060885d0, State: Initialized, Role: FOLLOWER
I20260924 19:26:38.118124 21683 consensus_queue.cc:260] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.118285 21683 ts_tablet_manager.cc:1428] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:38.118685 21685 tablet_bootstrap.cc:492] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: No bootstrap required, opened a new log
I20260924 19:26:38.118769 21685 ts_tablet_manager.cc:1397] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:38.118901 21685 raft_consensus.cc:348] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.118932 21684 tablet_bootstrap.cc:492] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: No bootstrap required, opened a new log
I20260924 19:26:38.118986 21685 raft_consensus.cc:374] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:38.118999 21684 ts_tablet_manager.cc:1397] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:38.119028 21685 raft_consensus.cc:729] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Initialized, Role: FOLLOWER
I20260924 19:26:38.119091 21685 consensus_queue.cc:260] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.119146 21684 raft_consensus.cc:348] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.119257 21684 raft_consensus.cc:374] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:38.119261 21685 ts_tablet_manager.cc:1428] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:38.119289 21684 raft_consensus.cc:729] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Initialized, Role: FOLLOWER
I20260924 19:26:38.119354 21684 consensus_queue.cc:260] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.119455 21684 ts_tablet_manager.cc:1428] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:38.399999 21690 raft_consensus.cc:482] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:38.400151 21690 raft_consensus.cc:504] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.400321 21690 leader_election.cc:288] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:38.400518 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22c17b91692c4f3a8a6be334bd48a69e" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2b3856d81242ca8ab5116335cfc71c" is_pre_election: true
I20260924 19:26:38.400514 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22c17b91692c4f3a8a6be334bd48a69e" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0" is_pre_election: true
I20260924 19:26:38.400606 21528 raft_consensus.cc:2454] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 0.
I20260924 19:26:38.400630 21248 raft_consensus.cc:2454] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 0.
I20260924 19:26:38.400821 21322 leader_election.cc:302] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, cd4fec8db3684ad987edfe541e02fa97; no voters:
I20260924 19:26:38.400954 21690 raft_consensus.cc:2781] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:38.401010 21690 raft_consensus.cc:482] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:38.401026 21690 raft_consensus.cc:3037] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:38.401784 21690 raft_consensus.cc:504] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.401938 21690 leader_election.cc:288] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 060109aa68604959a544bac8060885d0 (127.18.229.129:40547), 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:38.402073 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22c17b91692c4f3a8a6be334bd48a69e" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2b3856d81242ca8ab5116335cfc71c"
I20260924 19:26:38.402098 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22c17b91692c4f3a8a6be334bd48a69e" candidate_uuid: "cd4fec8db3684ad987edfe541e02fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060109aa68604959a544bac8060885d0"
I20260924 19:26:38.402148 21528 raft_consensus.cc:3037] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:38.402165 21248 raft_consensus.cc:3037] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:38.402781 21528 raft_consensus.cc:2454] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 1.
I20260924 19:26:38.402894 21248 raft_consensus.cc:2454] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd4fec8db3684ad987edfe541e02fa97 in term 1.
I20260924 19:26:38.402976 21319 leader_election.cc:302] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2b3856d81242ca8ab5116335cfc71c, cd4fec8db3684ad987edfe541e02fa97; no voters:
I20260924 19:26:38.403056 21690 raft_consensus.cc:2781] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:38.403203 21690 raft_consensus.cc:686] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 1 LEADER]: Becoming Leader. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Running, Role: LEADER
I20260924 19:26:38.403294 21690 consensus_queue.cc:237] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.403895 21089 catalog_manager.cc:5495] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "cd4fec8db3684ad987edfe541e02fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:26:38.474100 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.471804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.18.229.130:42247)
0924 19:26:38.471996 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.471998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.472006 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:38.472028 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.472028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.472040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.472137 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:38.472264 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.472759 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.473506 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.473693 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.473745 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.473867 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:38.473894 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:38.473934 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.473984 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.474001 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260924 19:26:38.476320 21528 raft_consensus.cc:1261] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Refusing update from remote peer cd4fec8db3684ad987edfe541e02fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:38.476339 21248 raft_consensus.cc:1261] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Refusing update from remote peer cd4fec8db3684ad987edfe541e02fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:38.476564 21690 consensus_queue.cc:1035] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:38.476692 21691 consensus_queue.cc:1035] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:38.486898 21089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:38.486979 21089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:38.487534 21089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Sending DeleteTablet for 3 replicas of tablet 22c17b91692c4f3a8a6be334bd48a69e
I20260924 19:26:38.487761 21228 tablet_service.cc:1514] Processing DeleteTablet for tablet 22c17b91692c4f3a8a6be334bd48a69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:38 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40902
I20260924 19:26:38.487779 21368 tablet_service.cc:1514] Processing DeleteTablet for tablet 22c17b91692c4f3a8a6be334bd48a69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:38 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:52978
I20260924 19:26:38.487874 21673 tablet_replica.cc:330] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: stopping tablet replica
I20260924 19:26:38.487882 21672 tablet_replica.cc:330] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: stopping tablet replica
I20260924 19:26:38.487931 21673 raft_consensus.cc:2229] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:38.487934 21672 raft_consensus.cc:2229] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:38.487977 21673 raft_consensus.cc:2258] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:38.487990 21672 raft_consensus.cc:2258] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:38.488093 21508 tablet_service.cc:1514] Processing DeleteTablet for tablet 22c17b91692c4f3a8a6be334bd48a69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:38 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40440
I20260924 19:26:38.488232 21089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:26:38.488303 21672 ts_tablet_manager.cc:1905] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:26:38.488384 21089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:38.488758 21673 ts_tablet_manager.cc:1905] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:38.489506 21228 tablet_service.cc:1467] Processing CreateTablet for tablet 7220817611e94596b0cf773624661877 (DEFAULT_TABLE table=test-table [id=39c128f8efc04309a02ad611c865d9c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:38.489558 21368 tablet_service.cc:1467] Processing CreateTablet for tablet 7220817611e94596b0cf773624661877 (DEFAULT_TABLE table=test-table [id=39c128f8efc04309a02ad611c865d9c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:38.489629 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7220817611e94596b0cf773624661877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:38.489641 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7220817611e94596b0cf773624661877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:38.489903 21672 ts_tablet_manager.cc:1918] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:38.489949 21672 log.cc:1196] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/22c17b91692c4f3a8a6be334bd48a69e
I20260924 19:26:38.490154 21672 ts_tablet_manager.cc:1939] T 22c17b91692c4f3a8a6be334bd48a69e P cd4fec8db3684ad987edfe541e02fa97: Deleting consensus metadata
I20260924 19:26:38.490272 21674 tablet_replica.cc:330] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: stopping tablet replica
I20260924 19:26:38.490324 21674 raft_consensus.cc:2229] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:38.490356 21674 raft_consensus.cc:2258] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:38.490489 21673 ts_tablet_manager.cc:1918] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:38.490536 21673 log.cc:1196] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/22c17b91692c4f3a8a6be334bd48a69e
I20260924 19:26:38.490743 21673 ts_tablet_manager.cc:1939] T 22c17b91692c4f3a8a6be334bd48a69e P 060109aa68604959a544bac8060885d0: Deleting consensus metadata
I20260924 19:26:38.490782 21674 ts_tablet_manager.cc:1905] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:38.490787 21685 tablet_bootstrap.cc:492] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Bootstrap starting.
I20260924 19:26:38.490877 21683 tablet_bootstrap.cc:492] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0: Bootstrap starting.
I20260924 19:26:38.491091 21685 tablet_bootstrap.cc:654] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:38.491400 21683 tablet_bootstrap.cc:654] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:38.491425 21077 catalog_manager.cc:4841] TS cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247): tablet 22c17b91692c4f3a8a6be334bd48a69e (table test-table [id=87bda96dd8164f7db71a17b78afef0b2]) successfully deleted
I20260924 19:26:38.491693 21508 tablet_service.cc:1467] Processing CreateTablet for tablet 7220817611e94596b0cf773624661877 (DEFAULT_TABLE table=test-table [id=39c128f8efc04309a02ad611c865d9c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:38.491791 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7220817611e94596b0cf773624661877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:38.491918 21674 ts_tablet_manager.cc:1918] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:26:38.492038 21683 tablet_bootstrap.cc:492] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0: No bootstrap required, opened a new log
I20260924 19:26:38.492069 21674 log.cc:1196] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/22c17b91692c4f3a8a6be334bd48a69e
I20260924 19:26:38.492082 21683 ts_tablet_manager.cc:1397] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:38.492229 21078 catalog_manager.cc:4841] TS 060109aa68604959a544bac8060885d0 (127.18.229.129:40547): tablet 22c17b91692c4f3a8a6be334bd48a69e (table test-table [id=87bda96dd8164f7db71a17b78afef0b2]) successfully deleted
I20260924 19:26:38.492252 21683 raft_consensus.cc:348] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.492311 21683 raft_consensus.cc:374] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:38.492329 21683 raft_consensus.cc:729] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060109aa68604959a544bac8060885d0, State: Initialized, Role: FOLLOWER
I20260924 19:26:38.492326 21674 ts_tablet_manager.cc:1939] T 22c17b91692c4f3a8a6be334bd48a69e P 8a2b3856d81242ca8ab5116335cfc71c: Deleting consensus metadata
I20260924 19:26:38.492378 21683 consensus_queue.cc:260] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.492453 21683 ts_tablet_manager.cc:1428] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:38.493227 21684 tablet_bootstrap.cc:492] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Bootstrap starting.
I20260924 19:26:38.493304 21685 tablet_bootstrap.cc:492] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: No bootstrap required, opened a new log
I20260924 19:26:38.493335 21685 ts_tablet_manager.cc:1397] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:26:38.493404 21685 raft_consensus.cc:348] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.493445 21685 raft_consensus.cc:374] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:38.493458 21685 raft_consensus.cc:729] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd4fec8db3684ad987edfe541e02fa97, State: Initialized, Role: FOLLOWER
I20260924 19:26:38.493461 21684 tablet_bootstrap.cc:654] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:38.493522 21685 consensus_queue.cc:260] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.493608 21685 ts_tablet_manager.cc:1428] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:38.493820 21075 catalog_manager.cc:4841] TS 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465): tablet 22c17b91692c4f3a8a6be334bd48a69e (table test-table [id=87bda96dd8164f7db71a17b78afef0b2]) successfully deleted
I20260924 19:26:38.494140 21684 tablet_bootstrap.cc:492] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: No bootstrap required, opened a new log
I20260924 19:26:38.494194 21684 ts_tablet_manager.cc:1397] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:26:38.494306 21684 raft_consensus.cc:348] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.494357 21684 raft_consensus.cc:374] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:38.494376 21684 raft_consensus.cc:729] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b3856d81242ca8ab5116335cfc71c, State: Initialized, Role: FOLLOWER
I20260924 19:26:38.494428 21684 consensus_queue.cc:260] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.494519 21684 ts_tablet_manager.cc:1428] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:38.514148 21689 raft_consensus.cc:482] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:38.514230 21689 raft_consensus.cc:504] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.514600 21689 leader_election.cc:288] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247), 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:38.517659 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.514813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:45445 (local address 127.18.229.130:42247)
0924 19:26:38.514858 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.514870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.514994 (+ 124us) server_negotiation.cc:407] Connection header received
0924 19:26:38.515045 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.515046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.515057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.515105 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:38.515281 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.515856 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.516925 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.517297 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.517353 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.517384 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.517551 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.517590 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:26:38.517688 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.514843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:58277 (local address 127.18.229.131:41465)
0924 19:26:38.514977 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.514980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.514990 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:38.515058 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.515059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.515067 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.515143 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:38.515317 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.515839 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.516932 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.517302 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.517397 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.517405 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.517582 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.517621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260924 19:26:38.517697 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.514743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:42247 (local address 127.18.229.129:45445)
0924 19:26:38.514881 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:38.514894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:38.514989 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:38.515123 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:38.515124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:38.515127 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:38.515205 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.515207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.516015 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.516016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.516813 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.516815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.517394 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.517395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.517466 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.517470 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:38.517523 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:26:38.517583 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260924 19:26:38.517704 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.514743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.18.229.129:58277)
0924 19:26:38.514923 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:38.514934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:38.514995 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:38.515151 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:38.515152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:38.515154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:38.515229 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.515230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.515962 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.515963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.516811 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.516813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.517409 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.517410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.517486 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.517489 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:38.517561 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:26:38.517599 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260924 19:26:38.517968 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220817611e94596b0cf773624661877" candidate_uuid: "060109aa68604959a544bac8060885d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4fec8db3684ad987edfe541e02fa97" is_pre_election: true
I20260924 19:26:38.518055 21388 raft_consensus.cc:2454] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060109aa68604959a544bac8060885d0 in term 0.
I20260924 19:26:38.518217 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220817611e94596b0cf773624661877" candidate_uuid: "060109aa68604959a544bac8060885d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2b3856d81242ca8ab5116335cfc71c" is_pre_election: true
I20260924 19:26:38.518311 21528 raft_consensus.cc:2454] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060109aa68604959a544bac8060885d0 in term 0.
I20260924 19:26:38.518296 21181 leader_election.cc:302] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, cd4fec8db3684ad987edfe541e02fa97; no voters:
I20260924 19:26:38.518443 21689 raft_consensus.cc:2781] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:38.518482 21689 raft_consensus.cc:482] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:38.518503 21689 raft_consensus.cc:3037] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:38.519166 21689 raft_consensus.cc:504] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.519307 21689 leader_election.cc:288] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [CANDIDATE]: Term 1 election: Requested vote from peers cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247), 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465)
I20260924 19:26:38.519433 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220817611e94596b0cf773624661877" candidate_uuid: "060109aa68604959a544bac8060885d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd4fec8db3684ad987edfe541e02fa97"
I20260924 19:26:38.519495 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220817611e94596b0cf773624661877" candidate_uuid: "060109aa68604959a544bac8060885d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2b3856d81242ca8ab5116335cfc71c"
I20260924 19:26:38.519536 21388 raft_consensus.cc:3037] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:38.519562 21528 raft_consensus.cc:3037] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:38.520279 21388 raft_consensus.cc:2454] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060109aa68604959a544bac8060885d0 in term 1.
I20260924 19:26:38.520280 21528 raft_consensus.cc:2454] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060109aa68604959a544bac8060885d0 in term 1.
I20260924 19:26:38.520483 21179 leader_election.cc:302] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060109aa68604959a544bac8060885d0, 8a2b3856d81242ca8ab5116335cfc71c; no voters:
I20260924 19:26:38.520578 21689 raft_consensus.cc:2781] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:38.520758 21689 raft_consensus.cc:686] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [term 1 LEADER]: Becoming Leader. State: Replica: 060109aa68604959a544bac8060885d0, State: Running, Role: LEADER
I20260924 19:26:38.520846 21689 consensus_queue.cc:237] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } }
I20260924 19:26:38.521495 21089 catalog_manager.cc:5495] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 060109aa68604959a544bac8060885d0 (127.18.229.129). New cstate: current_term: 1 leader_uuid: "060109aa68604959a544bac8060885d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060109aa68604959a544bac8060885d0" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:26:38.532367 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.530222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.18.229.129:40547)
0924 19:26:38.530372 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.530374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.530388 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:26:38.530418 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.530418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.530436 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.530503 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:38.530637 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.531099 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.531738 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.531931 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.531980 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.532092 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:38.532200 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:38.532250 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.532292 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.532306 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260924 19:26:38.536607 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.534610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:34739 (local address 127.18.229.131:41465)
0924 19:26:38.534666 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.534668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.534697 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:26:38.534753 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.534754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.534761 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.534805 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:38.534919 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.535337 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.535983 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.536222 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.536289 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.536450 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:38.536474 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:38.536507 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.536543 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.536559 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:26:38.536623 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.534480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41465 (local address 127.18.229.129:34739)
0924 19:26:38.534651 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:38.534660 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:38.534709 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:38.534818 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:38.534818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:38.534820 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:38.534867 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.534868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.535396 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.535397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.535920 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.535922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.536310 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.536310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.536371 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.536383 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:38.536513 (+ 130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:38.536514 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:26:38.536543 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 19:26:38.536570 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260924 19:26:38.538383 21528 raft_consensus.cc:1261] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Refusing update from remote peer 060109aa68604959a544bac8060885d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:38.538409 21388 raft_consensus.cc:1261] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Refusing update from remote peer 060109aa68604959a544bac8060885d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:38.538656 21705 consensus_queue.cc:1035] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd4fec8db3684ad987edfe541e02fa97" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:38.538761 21689 consensus_queue.cc:1035] T 7220817611e94596b0cf773624661877 P 060109aa68604959a544bac8060885d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2b3856d81242ca8ab5116335cfc71c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:38.545614 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.543212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:47733 (local address 127.18.229.129:40547)
0924 19:26:38.543285 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:38.543288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:38.543336 (+ 48us) server_negotiation.cc:407] Connection header received
0924 19:26:38.543401 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:38.543401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:38.543408 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:38.543447 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:38.543529 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.544044 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.544911 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.545178 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.545260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.545404 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:38.545433 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:38.545480 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:38.545537 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:26:38.545555 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:26:38.545651 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:38.543171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40547 (local address 127.18.229.131:47733)
0924 19:26:38.543283 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:38.543294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:38.543350 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:38.543455 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:38.543456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:38.543457 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:26:38.543495 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.543496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.544104 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.544104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.544826 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:38.544828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:38.545262 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:38.545263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:38.545329 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:38.545332 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:26:38.545479 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:26:38.545480 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:26:38.545530 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:26:38.545572 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":75,"threads_started":1}
W20260924 19:26:38.545979 21656 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:26:40.068466 21090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:40.068558 21090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:40.069362 21090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eaa0d0de5434f9ebdc9afe23b112fa7: Sending DeleteTablet for 3 replicas of tablet 7220817611e94596b0cf773624661877
I20260924 19:26:40.069636 21508 tablet_service.cc:1514] Processing DeleteTablet for tablet 7220817611e94596b0cf773624661877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:40 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40440
I20260924 19:26:40.069639 21368 tablet_service.cc:1514] Processing DeleteTablet for tablet 7220817611e94596b0cf773624661877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:40 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:52978
I20260924 19:26:40.069754 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21163
I20260924 19:26:40.069849 21742 tablet_replica.cc:330] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: stopping tablet replica
I20260924 19:26:40.069831 21228 tablet_service.cc:1514] Processing DeleteTablet for tablet 7220817611e94596b0cf773624661877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:40 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40902
I20260924 19:26:40.070278 21742 raft_consensus.cc:2229] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:40.070323 21742 raft_consensus.cc:2258] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:40.070448 21741 tablet_replica.cc:330] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: stopping tablet replica
I20260924 19:26:40.070505 21741 raft_consensus.cc:2229] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:40.070535 21741 raft_consensus.cc:2258] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:40.070580 21742 ts_tablet_manager.cc:1905] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:40.070760 21741 ts_tablet_manager.cc:1905] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:40.072010 21741 ts_tablet_manager.cc:1918] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:26:40.072050 21741 log.cc:1196] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/7220817611e94596b0cf773624661877
I20260924 19:26:40.072304 21741 ts_tablet_manager.cc:1939] T 7220817611e94596b0cf773624661877 P 8a2b3856d81242ca8ab5116335cfc71c: Deleting consensus metadata
I20260924 19:26:40.072592 21075 catalog_manager.cc:4841] TS 8a2b3856d81242ca8ab5116335cfc71c (127.18.229.131:41465): tablet 7220817611e94596b0cf773624661877 (table test-table [id=39c128f8efc04309a02ad611c865d9c0]) successfully deleted
I20260924 19:26:40.072628 21742 ts_tablet_manager.cc:1918] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:26:40.072681 21742 log.cc:1196] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/7220817611e94596b0cf773624661877
I20260924 19:26:40.072949 21742 ts_tablet_manager.cc:1939] T 7220817611e94596b0cf773624661877 P cd4fec8db3684ad987edfe541e02fa97: Deleting consensus metadata
I20260924 19:26:40.073243 21077 catalog_manager.cc:4841] TS cd4fec8db3684ad987edfe541e02fa97 (127.18.229.130:42247): tablet 7220817611e94596b0cf773624661877 (table test-table [id=39c128f8efc04309a02ad611c865d9c0]) successfully deleted
W20260924 19:26:40.076999 21078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:40547 (error 108)
I20260924 19:26:40.077205 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21303
I20260924 19:26:40.080322 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:40.077386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40547 (local address 127.0.0.1:43754)
0924 19:26:40.080218 (+ 2832us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:40.080273 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2804,"thread_start_us":58,"threads_started":1}
W20260924 19:26:40.080471 21078 catalog_manager.cc:4584] TS 060109aa68604959a544bac8060885d0 (127.18.229.129:40547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7220817611e94596b0cf773624661877: Network error: Client connection negotiation failed: client connection to 127.18.229.129:40547: connect: Connection refused (error 111)
I20260924 19:26:40.083416 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21443
I20260924 19:26:40.088882 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21059
2026-09-24T19:26:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5758 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:40 dist-test-slave-r8cc krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:40 dist-test-slave-r8cc krb5kdc[21747](info): set up 2 sockets
Sep 24 19:26:40 dist-test-slave-r8cc krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260924 19:26:41.107911 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:41.218827 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:39433
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:39433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:41.298573 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:41.298786 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:41.298820 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:41.299949 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:41.300019 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:41.300045 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:41.300076 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:41.300096 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:41.301573 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:39433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:41.301852 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:41.302080 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:41.307017 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:41.307004 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:41.307071 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:41.307054 21760 server_base.cc:1034] running on GCE node
I20260924 19:26:41.307574 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:41.307862 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:41.309002 21760 hybrid_clock.cc:648] HybridClock initialized: now 1790278001308993 us; error 33 us; skew 500 ppm
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:41.311439 21760 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:41.311899 21760 webserver.cc:460] Webserver started at http://127.18.229.190:45091/ using document root <none> and password file <none>
I20260924 19:26:41.312196 21760 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:41.312263 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:41.312400 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:41.313508 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "691b4a0992aa4badaa96496d19d1f8d7"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "99428887a138083152987958d8eec8ba"
server_key_iv: "6a534793f868a7d22cfc8bed066f5476"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.313862 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "691b4a0992aa4badaa96496d19d1f8d7"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "99428887a138083152987958d8eec8ba"
server_key_iv: "6a534793f868a7d22cfc8bed066f5476"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.315128 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:41.315801 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.315970 21760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:41.316071 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "691b4a0992aa4badaa96496d19d1f8d7"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "99428887a138083152987958d8eec8ba"
server_key_iv: "6a534793f868a7d22cfc8bed066f5476"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.316179 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:41.331590 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:41.333580 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:41.333742 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:41.337579 21760 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:39433
I20260924 19:26:41.337626 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:39433 every 8 connection(s)
I20260924 19:26:41.338012 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:41.338930 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:41.341261 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7: Bootstrap starting.
I20260924 19:26:41.341779 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:41.342027 21828 log.cc:824] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:41.342646 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7: No bootstrap required, opened a new log
I20260924 19:26:41.343497 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21760
I20260924 19:26:41.343600 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:41.343770 19350 external_mini_cluster.cc:1428] Setting key b368a2ad8b12221b78b25372f2c4e290
I20260924 19:26:41.344652 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691b4a0992aa4badaa96496d19d1f8d7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39433 } }
I20260924 19:26:41.344835 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:41.344870 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691b4a0992aa4badaa96496d19d1f8d7, State: Initialized, Role: FOLLOWER
I20260924 19:26:41.345005 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691b4a0992aa4badaa96496d19d1f8d7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39433 } }
I20260924 19:26:41.345088 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:41.345120 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:41.345166 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:41.346278 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691b4a0992aa4badaa96496d19d1f8d7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39433 } }
I20260924 19:26:41.346385 21828 leader_election.cc:302] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691b4a0992aa4badaa96496d19d1f8d7; no voters:
I20260924 19:26:41.346529 21828 leader_election.cc:288] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:41.346598 21833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:41.346731 21833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [term 1 LEADER]: Becoming Leader. State: Replica: 691b4a0992aa4badaa96496d19d1f8d7, State: Running, Role: LEADER
I20260924 19:26:41.346751 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:41.346870 21833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691b4a0992aa4badaa96496d19d1f8d7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39433 } }
I20260924 19:26:41.347240 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691b4a0992aa4badaa96496d19d1f8d7. Latest consensus state: current_term: 1 leader_uuid: "691b4a0992aa4badaa96496d19d1f8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691b4a0992aa4badaa96496d19d1f8d7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39433 } } }
I20260924 19:26:41.347373 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:41.347627 21842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:41.348321 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691b4a0992aa4badaa96496d19d1f8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691b4a0992aa4badaa96496d19d1f8d7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39433 } } }
I20260924 19:26:41.348366 21842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:41.348443 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:41.350248 21842 catalog_manager.cc:1347] Generated new cluster ID: 1672cdc95ec745dda2b238a849c70310
I20260924 19:26:41.350306 21842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:41.355350 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.344402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.18.229.190:39433)
0924 19:26:41.344614 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:41.344625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:41.344658 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:26:41.344770 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:41.344777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:41.344835 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:41.344980 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:41.345715 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.347130 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.348245 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.348844 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.348959 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.351802 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:41.351806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:41.351806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:41.351817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:41.353757 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.354170 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.354173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.354173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.354283 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.354496 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.354497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.354497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.354762 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:41.354915 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:41.355050 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:26:41.355166 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260924 19:26:41.364400 21842 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:41.364979 21842 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:41.372246 21842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691b4a0992aa4badaa96496d19d1f8d7: Generated new TSK 0
I20260924 19:26:41.372433 21842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:41.398034 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39433
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:41.477296 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:41.477526 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:41.477550 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:41.478667 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:41.478701 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:41.478727 21854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:41.480093 21854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:41.480310 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:41.480479 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:41.485270 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:41.485260 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:41.485258 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:41.485538 21854 server_base.cc:1034] running on GCE node
I20260924 19:26:41.485791 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:41.486055 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:41.487201 21854 hybrid_clock.cc:648] HybridClock initialized: now 1790278001487181 us; error 36 us; skew 500 ppm
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:41.489699 21854 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:41.490151 21854 webserver.cc:460] Webserver started at http://127.18.229.129:42465/ using document root <none> and password file <none>
I20260924 19:26:41.490394 21854 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:41.490464 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:41.490618 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:41.491699 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "29fdfb22a9974dcab726e2edbf2e0197"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "2a6f2b54a4aa75964bfd40af304485a2"
server_key_iv: "fb7d0b4e84d14d2b57cb7d1bc5506719"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.492146 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "29fdfb22a9974dcab726e2edbf2e0197"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "2a6f2b54a4aa75964bfd40af304485a2"
server_key_iv: "fb7d0b4e84d14d2b57cb7d1bc5506719"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.493479 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:41.494195 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.494367 21854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.494464 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "29fdfb22a9974dcab726e2edbf2e0197"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "2a6f2b54a4aa75964bfd40af304485a2"
server_key_iv: "fb7d0b4e84d14d2b57cb7d1bc5506719"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.494550 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:41.513854 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:41.516527 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:41.516680 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:41.516939 21854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:41.517266 21854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:41.517330 21854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.517372 21854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:41.517400 21854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.523594 21854 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:34053
I20260924 19:26:41.523635 21982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:34053 every 8 connection(s)
I20260924 19:26:41.523954 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:41.532657 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21854
I20260924 19:26:41.532763 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:41.532855 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.524796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:59721 (local address 127.18.229.190:39433)
0924 19:26:41.524851 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:41.524853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:41.525046 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:26:41.525522 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:41.525523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:41.525535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:41.525601 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:41.526510 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.526999 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.527943 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.528159 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.528206 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.530265 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:41.530266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:41.530267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:41.530273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:41.531509 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.531866 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.531866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.531867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.531912 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.532211 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.532211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.532211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.532308 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:41.532368 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:41.532651 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:26:41.532743 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:26:41.532959 19350 external_mini_cluster.cc:1428] Setting key 0045017e8e805fbc61d76a851a6eaf88
I20260924 19:26:41.533047 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.524750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:39433 (local address 127.18.229.129:59721)
0924 19:26:41.524914 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:41.524956 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:41.525442 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:41.525655 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:41.525658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:41.526239 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:41.526436 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:41.526440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.527076 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.527076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:41.527872 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:41.527874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.528215 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.528216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:41.528339 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.528873 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:41.528881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:41.530176 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:41.531577 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:41.531583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:41.531585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:41.531799 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:41.531980 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:41.531981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:41.531981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:41.532091 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:41.532363 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:41.532368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:41.532508 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:26:41.532634 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:26:41.532820 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260924 19:26:41.533622 21983 heartbeater.cc:347] Connected to a master server at 127.18.229.190:39433
I20260924 19:26:41.533725 21983 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:41.533913 21983 heartbeater.cc:511] Master 127.18.229.190:39433 requested a full tablet report, sending...
I20260924 19:26:41.534488 21792 ts_manager.cc:194] Registered new tserver with Master: 29fdfb22a9974dcab726e2edbf2e0197 (127.18.229.129:34053)
I20260924 19:26:41.535406 21792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:59721
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:41.555090 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39433
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:41.634895 21988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:41.635093 21988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:41.635116 21988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:41.636268 21988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:41.636302 21988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:41.636327 21988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:41.637665 21988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:41.637852 21988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:41.638025 21988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:41.643024 21994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:41.643029 21993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:41.643136 21988 server_base.cc:1034] running on GCE node
W20260924 19:26:41.643046 21996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:41.643479 21988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:41.643757 21988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:41.644927 21988 hybrid_clock.cc:648] HybridClock initialized: now 1790278001644922 us; error 37 us; skew 500 ppm
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:41.647429 21988 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:41.647905 21988 webserver.cc:460] Webserver started at http://127.18.229.130:39561/ using document root <none> and password file <none>
I20260924 19:26:41.648195 21988 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:41.648275 21988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:41.648444 21988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:41.649621 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "7cd719e8edc6430ea16de0a5976a949b"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "bdaae8ca28d9cd00f31aef58734f86f3"
server_key_iv: "3c2e61cb744a0fe79e88e5fa6e973563"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.650027 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "7cd719e8edc6430ea16de0a5976a949b"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "bdaae8ca28d9cd00f31aef58734f86f3"
server_key_iv: "3c2e61cb744a0fe79e88e5fa6e973563"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.651345 21988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:41.652040 22003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.652225 21988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:41.652325 21988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "7cd719e8edc6430ea16de0a5976a949b"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "bdaae8ca28d9cd00f31aef58734f86f3"
server_key_iv: "3c2e61cb744a0fe79e88e5fa6e973563"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.652410 21988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:41.667467 21988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:41.669420 21988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:41.669588 21988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:41.669826 21988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:41.670147 21988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:41.670207 21988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.670246 21988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:41.670274 21988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.676170 21988 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:33675
I20260924 19:26:41.676220 22116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:33675 every 8 connection(s)
I20260924 19:26:41.676555 21988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:41.679812 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 21988
I20260924 19:26:41.679929 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:41.680130 19350 external_mini_cluster.cc:1428] Setting key 9780c2e002f3e72ad930c5725965acd9
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:41.687034 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.677320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:55881 (local address 127.18.229.190:39433)
0924 19:26:41.677366 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:41.677369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:41.677566 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:26:41.678046 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:41.678047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:41.678057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:41.678104 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:41.678996 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.679535 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.681009 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.681192 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.681247 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.683848 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:41.683850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:41.683850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:41.683859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:41.685650 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.686044 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.686045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.686046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.686115 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.686353 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.686354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.686355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.686509 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:41.686584 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:41.686836 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:26:41.686931 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:26:41.687068 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.677294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:39433 (local address 127.18.229.130:55881)
0924 19:26:41.677440 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:41.677475 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:41.677955 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:41.678163 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:41.678166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:41.678746 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:41.678924 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:41.678929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.679616 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.679638 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:41.680913 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:41.680917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.681274 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.681275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:41.681458 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.682076 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:41.682087 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:41.683741 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:41.685734 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:41.685740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:41.685742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:41.685966 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:41.686187 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:41.686187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:41.686188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:41.686280 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:41.686579 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:41.686583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:41.686693 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:26:41.686775 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:26:41.686909 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260924 19:26:41.687621 22117 heartbeater.cc:347] Connected to a master server at 127.18.229.190:39433
I20260924 19:26:41.687732 22117 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:41.687948 22117 heartbeater.cc:511] Master 127.18.229.190:39433 requested a full tablet report, sending...
I20260924 19:26:41.688372 21792 ts_manager.cc:194] Registered new tserver with Master: 7cd719e8edc6430ea16de0a5976a949b (127.18.229.130:33675)
I20260924 19:26:41.688748 21792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:55881
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:41.704918 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39433
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:41.784523 22122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:41.784778 22122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:41.784816 22122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:41.785991 22122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:41.786069 22122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:41.786111 22122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:41.787555 22122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:41.787824 22122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:41.788057 22122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:41.792984 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:41.793035 22122 server_base.cc:1034] running on GCE node
W20260924 19:26:41.792994 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:41.792994 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:41.793432 22122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:41.793680 22122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:41.794845 22122 hybrid_clock.cc:648] HybridClock initialized: now 1790278001794828 us; error 30 us; skew 500 ppm
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:41.797402 22122 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:41.797899 22122 webserver.cc:460] Webserver started at http://127.18.229.131:46237/ using document root <none> and password file <none>
I20260924 19:26:41.798079 22122 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:41.798120 22122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:41.798202 22122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:41.799324 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "bc1736ff929d42c481bc7bd7192959ec"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "dbbe4d6ddf8c0d5e23ba1a24fa01e530"
server_key_iv: "9f69e198422144533ea691c17a693ed6"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.799625 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "bc1736ff929d42c481bc7bd7192959ec"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "dbbe4d6ddf8c0d5e23ba1a24fa01e530"
server_key_iv: "9f69e198422144533ea691c17a693ed6"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.800966 22122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:41.801680 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.801855 22122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:41.801954 22122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "bc1736ff929d42c481bc7bd7192959ec"
format_stamp: "Formatted at 2026-09-24 19:26:41 on dist-test-slave-r8cc"
server_key: "dbbe4d6ddf8c0d5e23ba1a24fa01e530"
server_key_iv: "9f69e198422144533ea691c17a693ed6"
server_key_version: "encryptionkey@0"
I20260924 19:26:41.802031 22122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:41.816061 22122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:41.818619 22122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:41.818781 22122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:41.819021 22122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:41.819381 22122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:41.819430 22122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.819463 22122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:41.819492 22122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:41.825790 22122 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:35563
I20260924 19:26:41.825852 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:35563 every 8 connection(s)
I20260924 19:26:41.826206 22122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:41.829623 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22122
I20260924 19:26:41.829737 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:41.829933 19350 external_mini_cluster.cc:1428] Setting key f1946747f5a627740990300ed02bcf1a
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278001, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:41.835618 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.826892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:54471 (local address 127.18.229.190:39433)
0924 19:26:41.826938 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:41.826939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:41.827182 (+ 243us) server_negotiation.cc:407] Connection header received
0924 19:26:41.827686 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:41.827687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:41.827697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:41.827749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:41.828740 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.829242 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.830706 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.830996 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.831052 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.833303 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:41.833305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:41.833305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:41.833312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:41.834486 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.834847 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.834848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.834848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.834898 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:41.835114 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:41.835114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:41.835115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:41.835211 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:41.835260 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:41.835497 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 19:26:41.835555 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:26:41.835798 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:41.826889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:39433 (local address 127.18.229.131:54471)
0924 19:26:41.827032 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:41.827066 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:41.827596 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:41.827808 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:41.827811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:41.828453 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:41.828658 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:41.828663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.829322 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.829323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:41.830621 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:41.830624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:41.831076 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:41.831077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:41.831177 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:41.831727 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:41.831743 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:41.833204 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:41.834556 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:41.834562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:41.834563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:41.834775 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:41.834964 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:41.834965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:41.834965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:41.835060 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:41.835268 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:41.835272 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:41.835381 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 19:26:41.835469 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:26:41.835584 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260924 19:26:41.836330 22251 heartbeater.cc:347] Connected to a master server at 127.18.229.190:39433
I20260924 19:26:41.836458 22251 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:41.836647 22251 heartbeater.cc:511] Master 127.18.229.190:39433 requested a full tablet report, sending...
I20260924 19:26:41.837071 21792 ts_manager.cc:194] Registered new tserver with Master: bc1736ff929d42c481bc7bd7192959ec (127.18.229.131:35563)
I20260924 19:26:41.837623 21792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:54471
I20260924 19:26:41.842151 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:26:41.854928 21832 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:41.853949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.18.229.190:39433)
0924 19:26:41.854029 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:41.854031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:41.854133 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:26:41.854221 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:41.854222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:41.854235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:41.854305 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:41.854851 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59530: BlockingRecv error: recv got EOF from 127.0.0.1:59530 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:26:41.856016 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21854
I20260924 19:26:41.860670 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21988
I20260924 19:26:41.864854 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22122
I20260924 19:26:41.868840 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 21760
2026-09-24T19:26:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1778 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[22263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[22263](info): set up 2 sockets
Sep 24 19:26:41 dist-test-slave-r8cc krb5kdc[22263](info): commencing operation
krb5kdc: starting...
W20260924 19:26:42.895925 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:42 dist-test-slave-r8cc krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:43.006644 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42347
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:43.088243 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:43.088438 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:43.088459 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:43.089584 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:43.089620 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:43.089634 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:43.089648 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:43.089659 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:43.091069 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:43.091256 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:43.091431 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:43.096096 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:43.096238 22276 server_base.cc:1034] running on GCE node
W20260924 19:26:43.096057 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:43.096072 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:43.096618 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:43.096903 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:43.098044 22276 hybrid_clock.cc:648] HybridClock initialized: now 1790278003098026 us; error 39 us; skew 500 ppm
Sep 24 19:26:43 dist-test-slave-r8cc krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278003, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:43.100590 22276 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:43.101053 22276 webserver.cc:460] Webserver started at http://127.18.229.190:40897/ using document root <none> and password file <none>
I20260924 19:26:43.101276 22276 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:43.101348 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:43.101498 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:43.102672 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "986173e18c1d463083da29d13dd306c4"
format_stamp: "Formatted at 2026-09-24 19:26:43 on dist-test-slave-r8cc"
server_key: "b55752dd17cf18518d4498cc82416c30"
server_key_iv: "5bb060aaa02010943412ef2405e42971"
server_key_version: "encryptionkey@0"
I20260924 19:26:43.103089 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "986173e18c1d463083da29d13dd306c4"
format_stamp: "Formatted at 2026-09-24 19:26:43 on dist-test-slave-r8cc"
server_key: "b55752dd17cf18518d4498cc82416c30"
server_key_iv: "5bb060aaa02010943412ef2405e42971"
server_key_version: "encryptionkey@0"
I20260924 19:26:43.104436 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:43.105182 22291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:43.105376 22276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:43.105471 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "986173e18c1d463083da29d13dd306c4"
format_stamp: "Formatted at 2026-09-24 19:26:43 on dist-test-slave-r8cc"
server_key: "b55752dd17cf18518d4498cc82416c30"
server_key_iv: "5bb060aaa02010943412ef2405e42971"
server_key_version: "encryptionkey@0"
I20260924 19:26:43.105551 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:43.128259 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:43.130167 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:43.130342 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:43.134837 22276 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42347
I20260924 19:26:43.134873 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42347 every 8 connection(s)
I20260924 19:26:43.135398 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:43.136073 22344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:43.138484 22344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4: Bootstrap starting.
I20260924 19:26:43.139099 22344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:43.139412 22344 log.cc:824] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:43.140214 22344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4: No bootstrap required, opened a new log
I20260924 19:26:43.141405 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22276
I20260924 19:26:43.141484 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:43.141659 19350 external_mini_cluster.cc:1428] Setting key 9f7d78f73de5327ba76eb2e6a86b461a
I20260924 19:26:43.141955 22344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986173e18c1d463083da29d13dd306c4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42347 } }
I20260924 19:26:43.142181 22344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:43.142232 22344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986173e18c1d463083da29d13dd306c4, State: Initialized, Role: FOLLOWER
I20260924 19:26:43.142407 22344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986173e18c1d463083da29d13dd306c4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42347 } }
I20260924 19:26:43.142479 22344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:43.142501 22344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:43.142565 22344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:43.143285 22344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986173e18c1d463083da29d13dd306c4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42347 } }
I20260924 19:26:43.143421 22344 leader_election.cc:302] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 986173e18c1d463083da29d13dd306c4; no voters:
I20260924 19:26:43.143580 22344 leader_election.cc:288] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:43.143612 22349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:43.143718 22349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [term 1 LEADER]: Becoming Leader. State: Replica: 986173e18c1d463083da29d13dd306c4, State: Running, Role: LEADER
I20260924 19:26:43.143811 22349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986173e18c1d463083da29d13dd306c4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42347 } }
I20260924 19:26:43.143848 22344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:43.144145 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 986173e18c1d463083da29d13dd306c4. Latest consensus state: current_term: 1 leader_uuid: "986173e18c1d463083da29d13dd306c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986173e18c1d463083da29d13dd306c4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42347 } } }
I20260924 19:26:43.144259 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:43.144407 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "986173e18c1d463083da29d13dd306c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986173e18c1d463083da29d13dd306c4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42347 } } }
I20260924 19:26:43.144466 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:43.144740 22354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:43.145292 22354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:43.147804 22354 catalog_manager.cc:1347] Generated new cluster ID: e6cd91d1827a49bbb1f5b4c155eacfa3
I20260924 19:26:43.147861 22354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:43 dist-test-slave-r8cc krb5kdc[22263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:43.153131 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:43.142183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.18.229.190:42347)
0924 19:26:43.142839 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:43.142850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:43.142882 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:26:43.143004 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:43.143012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:43.143074 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:43.143243 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:43.144037 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:43.145632 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:43.146667 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:43.147272 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:43.147385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:43.149815 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:43.149821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:43.149822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:43.149837 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:43.151481 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:43.151869 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:43.151872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:43.151872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:43.151973 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:43.152246 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:43.152247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:43.152247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:43.152498 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:43.152662 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:43.152791 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:26:43.152931 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":549,"threads_started":1}
I20260924 19:26:43.154026 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:26:43.161931 22354 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:43.162667 22354 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:43.165716 22354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 986173e18c1d463083da29d13dd306c4: Generated new TSK 0
I20260924 19:26:43.165933 22354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:26:43.169750 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:43.165589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.18.229.190:42347)
0924 19:26:43.165622 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:43.165624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:43.166073 (+ 449us) server_negotiation.cc:407] Connection header received
0924 19:26:43.166104 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:43.166105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:43.166118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:43.166173 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:43.166546 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:43.167345 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:43.168125 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:43.168469 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:43.168536 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:43.169245 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:43.169246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:43.169247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:43.169502 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:43.169523 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:43.169530 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:43.169592 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:43.169647 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 19:26:43.169674 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:26:43.172314 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22276
2026-09-24T19:26:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1301 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:43 dist-test-slave-r8cc krb5kdc[22377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:43 dist-test-slave-r8cc krb5kdc[22377](info): set up 2 sockets
Sep 24 19:26:43 dist-test-slave-r8cc krb5kdc[22377](info): commencing operation
krb5kdc: starting...
W20260924 19:26:44.203909 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:44.330437 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38395
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:44.410215 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:44.410413 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:44.410435 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:44.411499 22390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:44.411532 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:44.411546 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:44.411559 22390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:44.411571 22390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:44.413076 22390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:44.413269 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:44.413432 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:44.418263 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:44.418222 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.418358 22390 server_base.cc:1034] running on GCE node
W20260924 19:26:44.418222 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.418711 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:44.418957 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:44.420095 22390 hybrid_clock.cc:648] HybridClock initialized: now 1790278004420079 us; error 29 us; skew 500 ppm
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:44.422426 22390 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:44.422881 22390 webserver.cc:460] Webserver started at http://127.18.229.190:44341/ using document root <none> and password file <none>
I20260924 19:26:44.423113 22390 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:44.423180 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:44.423334 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:44.424451 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "d3ae47bf707b4d27a162cd94ad02e88f"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "71eb73127c08e4e8e92f3e584307ecbc"
server_key_iv: "883b35c3bf9702c879a4def79b67ccdb"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.424801 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "d3ae47bf707b4d27a162cd94ad02e88f"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "71eb73127c08e4e8e92f3e584307ecbc"
server_key_iv: "883b35c3bf9702c879a4def79b67ccdb"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.426046 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:44.426738 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.426906 22390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:44.427002 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "d3ae47bf707b4d27a162cd94ad02e88f"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "71eb73127c08e4e8e92f3e584307ecbc"
server_key_iv: "883b35c3bf9702c879a4def79b67ccdb"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.427083 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:44.448819 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:44.450758 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:44.450925 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:44.455000 22390 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:38395
I20260924 19:26:44.455071 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:38395 every 8 connection(s)
I20260924 19:26:44.455444 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:44.456491 22458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:44.458705 22458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f: Bootstrap starting.
I20260924 19:26:44.459215 22458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:44.459447 22458 log.cc:824] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:44.460075 22458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f: No bootstrap required, opened a new log
I20260924 19:26:44.461642 22458 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38395 } }
I20260924 19:26:44.461779 22458 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:44.461802 22458 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ae47bf707b4d27a162cd94ad02e88f, State: Initialized, Role: FOLLOWER
I20260924 19:26:44.461943 22458 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38395 } }
I20260924 19:26:44.462028 22458 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:44.462054 22458 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:44.462083 22458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:44.462821 22458 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38395 } }
I20260924 19:26:44.462905 22458 leader_election.cc:302] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3ae47bf707b4d27a162cd94ad02e88f; no voters:
I20260924 19:26:44.463022 22458 leader_election.cc:288] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:44.463073 22461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:44.463233 22461 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [term 1 LEADER]: Becoming Leader. State: Replica: d3ae47bf707b4d27a162cd94ad02e88f, State: Running, Role: LEADER
I20260924 19:26:44.463265 22458 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:44.463379 22461 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38395 } }
I20260924 19:26:44.463828 22463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3ae47bf707b4d27a162cd94ad02e88f. Latest consensus state: current_term: 1 leader_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38395 } } }
I20260924 19:26:44.463927 22463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:44.464146 22470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:44.464330 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ae47bf707b4d27a162cd94ad02e88f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38395 } } }
I20260924 19:26:44.464473 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:44.464656 22470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:44.465399 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22390
I20260924 19:26:44.465502 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:44.465678 19350 external_mini_cluster.cc:1428] Setting key 5bc159385622cec2c3051472692dc696
I20260924 19:26:44.466234 22470 catalog_manager.cc:1347] Generated new cluster ID: 2b56513ff6c24c8b91cc56eea8654342
I20260924 19:26:44.466281 22470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:44.473861 22470 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:44.474490 22470 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:44.477833 22470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3ae47bf707b4d27a162cd94ad02e88f: Generated new TSK 0
I20260924 19:26:44.478040 22470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:26:44.478976 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.466122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.18.229.190:38395)
0924 19:26:44.466284 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:44.466293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:44.468341 (+ 2048us) server_negotiation.cc:407] Connection header received
0924 19:26:44.468462 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:44.468470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:44.468527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:44.468681 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:44.469375 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.470767 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.471785 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.472427 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.472531 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.475370 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:44.475374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:44.475374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:44.475385 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:44.477319 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.477717 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.477721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.477721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.477844 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.478063 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.478065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.478065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.478334 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:44.478443 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:44.478577 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:26:44.478684 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:44.502556 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38395
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:44.582477 22484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:44.582666 22484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:44.582688 22484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:44.583819 22484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:44.583859 22484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:44.583885 22484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:44.585320 22484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:44.585548 22484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:44.585783 22484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:44.590718 22490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.590862 22484 server_base.cc:1034] running on GCE node
W20260924 19:26:44.590680 22492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:44.590703 22489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.591187 22484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:44.591432 22484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:44.592587 22484 hybrid_clock.cc:648] HybridClock initialized: now 1790278004592582 us; error 51 us; skew 500 ppm
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:44.594921 22484 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:44.595371 22484 webserver.cc:460] Webserver started at http://127.18.229.129:39119/ using document root <none> and password file <none>
I20260924 19:26:44.595589 22484 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:44.595654 22484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:44.595808 22484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:44.596947 22484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "40be1ddffd964cacbf4c45b2a41171d6"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "8367cafd8de1e01b9425fcedcfa76d26"
server_key_iv: "16b0d3b00b0fff5af159f6b8b1313a6c"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.597290 22484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "40be1ddffd964cacbf4c45b2a41171d6"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "8367cafd8de1e01b9425fcedcfa76d26"
server_key_iv: "16b0d3b00b0fff5af159f6b8b1313a6c"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.598538 22484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:44.599225 22499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.599398 22484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:44.599498 22484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "40be1ddffd964cacbf4c45b2a41171d6"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "8367cafd8de1e01b9425fcedcfa76d26"
server_key_iv: "16b0d3b00b0fff5af159f6b8b1313a6c"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.599570 22484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:44.616607 22484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:44.618346 22484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:44.618502 22484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:44.618722 22484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:44.619020 22484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:44.619072 22484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.619112 22484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:44.619140 22484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.625120 22484 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:42359
I20260924 19:26:44.625166 22612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:42359 every 8 connection(s)
I20260924 19:26:44.625465 22484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:44.627103 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22484
I20260924 19:26:44.627240 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:44.627435 19350 external_mini_cluster.cc:1428] Setting key a94de0d7a7cbca31be0fd6c7e58d470c
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
I20260924 19:26:44.635397 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.626097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:58545 (local address 127.18.229.190:38395)
0924 19:26:44.626137 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:44.626141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:44.626339 (+ 198us) server_negotiation.cc:407] Connection header received
0924 19:26:44.626850 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:44.626852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:44.626881 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:44.626945 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:44.628055 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.628607 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.629971 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.630170 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.630282 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.632628 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:44.632629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:44.632629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:44.632638 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:44.634021 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.634442 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.634444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.634444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.634503 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.634768 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.634769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.634770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.634881 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:44.634958 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:44.635214 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 19:26:44.635302 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:26:44.635560 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.626071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38395 (local address 127.18.229.129:58545)
0924 19:26:44.626217 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:44.626257 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:44.626742 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:44.627037 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:44.627046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:44.627767 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:44.627959 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:44.627964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.628700 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.628700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:44.629889 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:44.629892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.630303 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.630304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:44.630443 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.630972 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:44.630982 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:44.632535 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:44.634120 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:44.634127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:44.634129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:44.634351 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:44.634584 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:44.634585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:44.634585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:44.634693 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:44.634947 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:44.634953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:44.635087 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:26:44.635194 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:26:44.635340 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260924 19:26:44.636034 22613 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38395
I20260924 19:26:44.636157 22613 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:44.636350 22613 heartbeater.cc:511] Master 127.18.229.190:38395 requested a full tablet report, sending...
I20260924 19:26:44.636869 22422 ts_manager.cc:194] Registered new tserver with Master: 40be1ddffd964cacbf4c45b2a41171d6 (127.18.229.129:42359)
I20260924 19:26:44.637595 22422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:58545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:44.651031 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38395
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:44.730851 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:44.731053 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:44.731076 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:44.732234 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:44.732275 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:44.732335 22618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:44.733778 22618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:44.734035 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:44.734269 22618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:44.739219 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:44.739264 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:44.739248 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.739431 22618 server_base.cc:1034] running on GCE node
I20260924 19:26:44.739696 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:44.739945 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:44.741104 22618 hybrid_clock.cc:648] HybridClock initialized: now 1790278004741092 us; error 51 us; skew 500 ppm
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:44.743449 22618 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:44.743803 22618 webserver.cc:460] Webserver started at http://127.18.229.130:33695/ using document root <none> and password file <none>
I20260924 19:26:44.744028 22618 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:44.744071 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:44.744252 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:44.745246 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "abce0efe8be94a4995987753e2d05c99"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "ef66255ac7d0c419b8d31eaa1504211b"
server_key_iv: "d974d2cb002301e5c7ea911d4b38d124"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.745569 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "abce0efe8be94a4995987753e2d05c99"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "ef66255ac7d0c419b8d31eaa1504211b"
server_key_iv: "d974d2cb002301e5c7ea911d4b38d124"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.746851 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:26:44.747498 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.747690 22618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:44.747761 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "abce0efe8be94a4995987753e2d05c99"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "ef66255ac7d0c419b8d31eaa1504211b"
server_key_iv: "d974d2cb002301e5c7ea911d4b38d124"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.747844 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:44.761384 22618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:44.763098 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:44.763250 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:44.763528 22618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:44.763844 22618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:44.763875 22618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.763927 22618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:44.763955 22618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.769783 22618 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:43689
I20260924 19:26:44.769841 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:43689 every 8 connection(s)
I20260924 19:26:44.770124 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:44.775923 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22618
I20260924 19:26:44.776012 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:44.776224 19350 external_mini_cluster.cc:1428] Setting key c54c0f70edfaee3392f934803f2e0b31
I20260924 19:26:44.779109 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.770796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:58559 (local address 127.18.229.190:38395)
0924 19:26:44.770889 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:44.770891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:44.771088 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:26:44.771555 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:44.771556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:44.771565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:44.771623 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:44.772601 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.773095 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.774020 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.774195 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.774242 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.776359 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:44.776361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:44.776361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:44.776367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:44.777749 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.778166 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.778168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.778168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.778224 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.778476 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.778477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.778478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.778588 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:44.778664 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:44.778937 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 19:26:44.779015 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:26:44.779312 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.770746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38395 (local address 127.18.229.130:58559)
0924 19:26:44.770957 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:44.770997 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:44.771470 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:44.771692 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:44.771695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:44.772329 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:44.772518 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:44.772524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.773172 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.773172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:44.773951 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:44.773953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.774263 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.774263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:44.774358 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.774788 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:44.774796 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:44.776235 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:44.777842 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:44.777848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:44.777851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:44.778074 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:44.778291 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:44.778291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:44.778292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:44.778407 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:44.778655 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:44.778661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:44.778799 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 19:26:44.778917 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 19:26:44.779078 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260924 19:26:44.779871 22747 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38395
I20260924 19:26:44.779980 22747 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:44.780191 22747 heartbeater.cc:511] Master 127.18.229.190:38395 requested a full tablet report, sending...
I20260924 19:26:44.780587 22422 ts_manager.cc:194] Registered new tserver with Master: abce0efe8be94a4995987753e2d05c99 (127.18.229.130:43689)
I20260924 19:26:44.780972 22422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:58559
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:44.798517 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38395
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:44.878489 22752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:44.878703 22752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:44.878727 22752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:44.879920 22752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:44.879966 22752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:44.880028 22752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:44.881536 22752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:44.881798 22752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:44.882032 22752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:44.887081 22757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:44.887089 22760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.887136 22752 server_base.cc:1034] running on GCE node
W20260924 19:26:44.887099 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:44.887455 22752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:44.887656 22752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:44.888794 22752 hybrid_clock.cc:648] HybridClock initialized: now 1790278004888780 us; error 30 us; skew 500 ppm
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:44.891377 22752 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:44.891839 22752 webserver.cc:460] Webserver started at http://127.18.229.131:32869/ using document root <none> and password file <none>
I20260924 19:26:44.892032 22752 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:44.892073 22752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:44.892220 22752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:44.893291 22752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "396c5577ec9647a4bfbd13de47e67d16"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "9256aad8a7971c8c2adaa45f4c1c1fc0"
server_key_iv: "97d23545868894ec801381756ea1d766"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.893596 22752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "396c5577ec9647a4bfbd13de47e67d16"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "9256aad8a7971c8c2adaa45f4c1c1fc0"
server_key_iv: "97d23545868894ec801381756ea1d766"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.894797 22752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:44.895442 22767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.895612 22752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:44.895713 22752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "396c5577ec9647a4bfbd13de47e67d16"
format_stamp: "Formatted at 2026-09-24 19:26:44 on dist-test-slave-r8cc"
server_key: "9256aad8a7971c8c2adaa45f4c1c1fc0"
server_key_iv: "97d23545868894ec801381756ea1d766"
server_key_version: "encryptionkey@0"
I20260924 19:26:44.895787 22752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:44.910601 22752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:44.913178 22752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:44.913345 22752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:44.913578 22752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:44.913898 22752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:44.913952 22752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.913987 22752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:44.914016 22752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:44.919489 22752 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:45919
I20260924 19:26:44.919593 22880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:45919 every 8 connection(s)
I20260924 19:26:44.919881 22752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:44.923132 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22752
I20260924 19:26:44.923229 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:44.923434 19350 external_mini_cluster.cc:1428] Setting key b87c80f28dbd36a600f08e75663635ea
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:44.929324 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.920631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:46833 (local address 127.18.229.190:38395)
0924 19:26:44.920674 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:44.920678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:44.920892 (+ 214us) server_negotiation.cc:407] Connection header received
0924 19:26:44.921436 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:44.921437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:44.921451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:44.921500 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:44.922456 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.923035 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.924479 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.924676 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.924730 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.926749 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:44.926750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:44.926751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:44.926759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:44.928246 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.928586 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.928587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.928587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.928636 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.928836 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.928837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.928837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.928936 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:44.928986 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:44.929196 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:26:44.929252 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:26:44.929566 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.920605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38395 (local address 127.18.229.131:46833)
0924 19:26:44.920755 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:44.920799 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:44.921324 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:44.921569 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:44.921573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:44.922134 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:44.922343 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:44.922347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.923131 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.923132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:44.924401 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:44.924404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.924747 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.924748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:44.924857 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.925337 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:44.925345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:44.926663 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:44.928319 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:44.928323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:44.928325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:44.928524 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:44.928691 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:44.928692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:44.928692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:44.928783 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:44.928990 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:44.928993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:44.929104 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 19:26:44.929184 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:26:44.929299 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260924 19:26:44.930006 22881 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38395
I20260924 19:26:44.930086 22881 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:44.930225 22881 heartbeater.cc:511] Master 127.18.229.190:38395 requested a full tablet report, sending...
I20260924 19:26:44.930526 22422 ts_manager.cc:194] Registered new tserver with Master: 396c5577ec9647a4bfbd13de47e67d16 (127.18.229.131:45919)
I20260924 19:26:44.930907 22422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:46833
I20260924 19:26:44.935135 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 19:26:44 dist-test-slave-r8cc krb5kdc[22377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278004, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:44.957232 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:44.950390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.18.229.190:38395)
0924 19:26:44.950448 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:44.950451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:44.950546 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:26:44.950633 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:44.950634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:44.950643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:44.950702 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:44.951247 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.951824 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.952610 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:44.952803 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:44.952857 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:44.954998 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:44.955000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:44.955000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:44.955007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:44.956246 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.956562 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.956563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.956563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.956611 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:44.956795 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:44.956796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:44.956796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:44.956933 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:44.957005 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:44.957108 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:26:44.957166 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 19:26:44.957451 22422 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52416
I20260924 19:26:44.957897 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22484
I20260924 19:26:44.962396 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22618
I20260924 19:26:44.966604 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22752
I20260924 19:26:44.970633 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22390
2026-09-24T19:26:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1800 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:45 dist-test-slave-r8cc krb5kdc[22893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:45 dist-test-slave-r8cc krb5kdc[22893](info): set up 2 sockets
Sep 24 19:26:45 dist-test-slave-r8cc krb5kdc[22893](info): commencing operation
krb5kdc: starting...
W20260924 19:26:46.039793 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 19:26:46.249011 19350 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.209s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 19:26:46.415258 19350 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.166s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 19:26:46.617949 19350 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.203s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/test-user.keytab.
W20260924 19:26:47.216859 19350 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.599s user 0.000s sys 0.002s
Sep 24 19:26:47 dist-test-slave-r8cc krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
W20260924 19:26:48.630439 19350 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.229.190: real 1.393s user 0.000s sys 0.003s
I20260924 19:26:48.630789 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:33481
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:48.712384 22906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:48.712615 22906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:48.712688 22906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:48.713841 22906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:48.713905 22906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:48.713932 22906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:48.713963 22906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:48.713989 22906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:48.715555 22906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:33481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:48.715832 22906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:48.716053 22906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:48.720993 22911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:48.720999 22912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:48.721047 22906 server_base.cc:1034] running on GCE node
W20260924 19:26:48.721014 22914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:48.721417 22906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:48.721701 22906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:48.722844 22906 hybrid_clock.cc:648] HybridClock initialized: now 1790278008722832 us; error 28 us; skew 500 ppm
Sep 24 19:26:48 dist-test-slave-r8cc krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:48.730177 22906 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:48.730690 22906 webserver.cc:460] Webserver started at http://127.18.229.190:35033/ using document root <none> and password file <none>
I20260924 19:26:48.730892 22906 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:48.730933 22906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:48.731022 22906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:48.732024 22906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "e2e63edaf58d42bc9450cb96173e9a62"
format_stamp: "Formatted at 2026-09-24 19:26:48 on dist-test-slave-r8cc"
server_key: "53f0aecc26b70a7099b5bab3f2c4c9cd"
server_key_iv: "cf2dd39fd6eb406b09851cf10e0d1ca3"
server_key_version: "encryptionkey@0"
I20260924 19:26:48.732395 22906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "e2e63edaf58d42bc9450cb96173e9a62"
format_stamp: "Formatted at 2026-09-24 19:26:48 on dist-test-slave-r8cc"
server_key: "53f0aecc26b70a7099b5bab3f2c4c9cd"
server_key_iv: "cf2dd39fd6eb406b09851cf10e0d1ca3"
server_key_version: "encryptionkey@0"
I20260924 19:26:48.733626 22906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:48.734313 22921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:48.734459 22906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:48.734548 22906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "e2e63edaf58d42bc9450cb96173e9a62"
format_stamp: "Formatted at 2026-09-24 19:26:48 on dist-test-slave-r8cc"
server_key: "53f0aecc26b70a7099b5bab3f2c4c9cd"
server_key_iv: "cf2dd39fd6eb406b09851cf10e0d1ca3"
server_key_version: "encryptionkey@0"
I20260924 19:26:48.734601 22906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:48.748346 22906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:48.750137 22906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:48.750294 22906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:48.754195 22906 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:33481
I20260924 19:26:48.754261 22973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:33481 every 8 connection(s)
I20260924 19:26:48.754745 22906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:48.755445 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:48.755623 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 22906
I20260924 19:26:48.755728 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:48.755937 19350 external_mini_cluster.cc:1428] Setting key 79da84e60c9d205ab39f9099d8eee3e7
I20260924 19:26:48.758435 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62: Bootstrap starting.
I20260924 19:26:48.759021 22974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:48.759315 22974 log.cc:824] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:48.759958 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62: No bootstrap required, opened a new log
I20260924 19:26:48.761950 22974 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e63edaf58d42bc9450cb96173e9a62" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33481 } }
I20260924 19:26:48.762106 22974 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:48.762176 22974 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e63edaf58d42bc9450cb96173e9a62, State: Initialized, Role: FOLLOWER
Sep 24 19:26:48 dist-test-slave-r8cc krb5kdc[22893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:48.762349 22974 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e63edaf58d42bc9450cb96173e9a62" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33481 } }
I20260924 19:26:48.762468 22974 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:48.762516 22974 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:48.762557 22974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:48.763190 22974 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e63edaf58d42bc9450cb96173e9a62" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33481 } }
I20260924 19:26:48.763309 22974 leader_election.cc:302] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2e63edaf58d42bc9450cb96173e9a62; no voters:
I20260924 19:26:48.763499 22974 leader_election.cc:288] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:48.763525 22979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:48.763649 22979 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [term 1 LEADER]: Becoming Leader. State: Replica: e2e63edaf58d42bc9450cb96173e9a62, State: Running, Role: LEADER
I20260924 19:26:48.763772 22974 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:48.764027 22979 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e63edaf58d42bc9450cb96173e9a62" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33481 } }
I20260924 19:26:48.764405 22980 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2e63edaf58d42bc9450cb96173e9a62. Latest consensus state: current_term: 1 leader_uuid: "e2e63edaf58d42bc9450cb96173e9a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e63edaf58d42bc9450cb96173e9a62" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33481 } } }
I20260924 19:26:48.764496 22980 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:48.764716 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2e63edaf58d42bc9450cb96173e9a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e63edaf58d42bc9450cb96173e9a62" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33481 } } }
I20260924 19:26:48.765102 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62 [sys.catalog]: This master's current role is: LEADER
W20260924 19:26:48.765671 22995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:48.765726 22995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:48.765790 22987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:48.765959 22987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:48.766662 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:48.756539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.18.229.190:33481)
0924 19:26:48.756716 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:48.756724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:48.756747 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:26:48.757160 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:48.757166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:48.757211 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:48.757327 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:48.758109 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:48.759030 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:48.759882 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:48.760376 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:48.760505 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:48.762906 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:48.762910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:48.762910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:48.762919 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:48.764946 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:48.765358 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:48.765387 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:48.765388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:48.765509 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:48.765778 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:48.765779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:48.765780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:48.766088 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:48.766231 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:48.766362 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:26:48.766501 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260924 19:26:48.767452 22987 catalog_manager.cc:1347] Generated new cluster ID: 29bb3ea473ca40de9378854ac1653bf4
I20260924 19:26:48.767498 22987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:48.776417 22987 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:48.776959 22987 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:48.778844 22987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2e63edaf58d42bc9450cb96173e9a62: Generated new TSK 0
I20260924 19:26:48.779009 22987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:48.808102 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33481
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:48.914603 23000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:48.914812 23000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:48.914836 23000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:48.916003 23000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:48.916046 23000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:48.916072 23000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:48.917583 23000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:48.917837 23000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:48.918066 23000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:48.924129 23006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:48.924198 23008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:48.924301 23000 server_base.cc:1034] running on GCE node
W20260924 19:26:48.924379 23005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:48.924585 23000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:48.924837 23000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:48.926000 23000 hybrid_clock.cc:648] HybridClock initialized: now 1790278008925979 us; error 39 us; skew 500 ppm
Sep 24 19:26:48 dist-test-slave-r8cc krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:48.928568 23000 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:48.928998 23000 webserver.cc:460] Webserver started at http://127.18.229.129:43309/ using document root <none> and password file <none>
I20260924 19:26:48.929241 23000 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:48.929311 23000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:48.929463 23000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:48.930532 23000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "68e49f8347874d9891bac9ab9d35f96b"
format_stamp: "Formatted at 2026-09-24 19:26:48 on dist-test-slave-r8cc"
server_key: "a4c0a524e4ae4e41b9a2e11a81c4b1b2"
server_key_iv: "0c2844d6591829058f6bb0ea173e1898"
server_key_version: "encryptionkey@0"
I20260924 19:26:48.930891 23000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "68e49f8347874d9891bac9ab9d35f96b"
format_stamp: "Formatted at 2026-09-24 19:26:48 on dist-test-slave-r8cc"
server_key: "a4c0a524e4ae4e41b9a2e11a81c4b1b2"
server_key_iv: "0c2844d6591829058f6bb0ea173e1898"
server_key_version: "encryptionkey@0"
I20260924 19:26:48.932243 23000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:48.932969 23015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:48.933146 23000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:48.933243 23000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "68e49f8347874d9891bac9ab9d35f96b"
format_stamp: "Formatted at 2026-09-24 19:26:48 on dist-test-slave-r8cc"
server_key: "a4c0a524e4ae4e41b9a2e11a81c4b1b2"
server_key_iv: "0c2844d6591829058f6bb0ea173e1898"
server_key_version: "encryptionkey@0"
I20260924 19:26:48.933333 23000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:48.948920 23000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:48.951411 23000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:48.951584 23000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:48.951833 23000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:48.952178 23000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:48.952232 23000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:48.952266 23000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:48.952328 23000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:48.958129 23000 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:38925
I20260924 19:26:48.958186 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:38925 every 8 connection(s)
I20260924 19:26:48.958482 23000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:48.963711 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23000
I20260924 19:26:48.963845 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:48.964032 19350 external_mini_cluster.cc:1428] Setting key 8eea8f0ece84646b9388cb30abee9b98
Sep 24 19:26:48 dist-test-slave-r8cc krb5kdc[22893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:48.968742 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:48.959149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:57325 (local address 127.18.229.190:33481)
0924 19:26:48.959214 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:48.959216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:48.959411 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:26:48.960444 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:48.960446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:48.960457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:48.960524 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:48.961150 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:48.961644 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:48.962678 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:48.962887 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:48.962976 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:48.965515 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:48.965516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:48.965516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:48.965524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:48.967242 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:48.967616 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:48.967617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:48.967617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:48.967685 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:48.967909 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:48.967910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:48.967910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:48.968064 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:48.968162 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:48.968491 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 19:26:48.968588 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:26:48.968852 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:48.959146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33481 (local address 127.18.229.129:57325)
0924 19:26:48.959291 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:48.959327 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:48.960331 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:48.960618 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:48.960621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:48.960858 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:48.961041 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:48.961045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:48.961752 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:48.961753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:48.962568 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:48.962570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:48.963000 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:48.963001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:48.963124 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:48.963577 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:48.963585 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:48.965419 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:48.967323 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:48.967330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:48.967331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:48.967543 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:48.967751 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:48.967752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:48.967752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:48.967848 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:48.968206 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:48.968213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:48.968342 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 19:26:48.968441 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:26:48.968637 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 19:26:48.969396 23129 heartbeater.cc:347] Connected to a master server at 127.18.229.190:33481
I20260924 19:26:48.969511 23129 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:48.969687 23129 heartbeater.cc:511] Master 127.18.229.190:33481 requested a full tablet report, sending...
I20260924 19:26:48.970244 22938 ts_manager.cc:194] Registered new tserver with Master: 68e49f8347874d9891bac9ab9d35f96b (127.18.229.129:38925)
I20260924 19:26:48.970845 22938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:57325
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:48.989099 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33481
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:49.068465 23134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:49.068663 23134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:49.068686 23134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:49.069850 23134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:49.069892 23134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:49.069919 23134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:49.071530 23134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:49.071769 23134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:49.071993 23134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:49.076855 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:49.076869 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:49.077034 23139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:49.077086 23134 server_base.cc:1034] running on GCE node
I20260924 19:26:49.077332 23134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:49.077534 23134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:49.078665 23134 hybrid_clock.cc:648] HybridClock initialized: now 1790278009078655 us; error 30 us; skew 500 ppm
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:49.081074 23134 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:49.081524 23134 webserver.cc:460] Webserver started at http://127.18.229.130:44391/ using document root <none> and password file <none>
I20260924 19:26:49.081754 23134 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:49.081827 23134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:49.081975 23134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:49.082984 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "50bded30aafb4c6c9a17295cffd47bb5"
format_stamp: "Formatted at 2026-09-24 19:26:49 on dist-test-slave-r8cc"
server_key: "c3854ee4a22938508325e13f6c483c54"
server_key_iv: "301ba90ce1b82d276bd01ec03498647a"
server_key_version: "encryptionkey@0"
I20260924 19:26:49.083336 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "50bded30aafb4c6c9a17295cffd47bb5"
format_stamp: "Formatted at 2026-09-24 19:26:49 on dist-test-slave-r8cc"
server_key: "c3854ee4a22938508325e13f6c483c54"
server_key_iv: "301ba90ce1b82d276bd01ec03498647a"
server_key_version: "encryptionkey@0"
I20260924 19:26:49.084704 23134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:49.085410 23149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.085577 23134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.085675 23134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "50bded30aafb4c6c9a17295cffd47bb5"
format_stamp: "Formatted at 2026-09-24 19:26:49 on dist-test-slave-r8cc"
server_key: "c3854ee4a22938508325e13f6c483c54"
server_key_iv: "301ba90ce1b82d276bd01ec03498647a"
server_key_version: "encryptionkey@0"
I20260924 19:26:49.085749 23134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:49.104735 23134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:49.106573 23134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:49.106739 23134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:49.106981 23134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:49.107285 23134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:49.107339 23134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.107380 23134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:49.107409 23134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.113214 23134 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:35321
I20260924 19:26:49.113255 23262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:35321 every 8 connection(s)
I20260924 19:26:49.113566 23134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:49.113839 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23134
I20260924 19:26:49.113941 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:49.114135 19350 external_mini_cluster.cc:1428] Setting key e9af64ce8803127aa90fcb154662167e
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
I20260924 19:26:49.124229 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:49.114303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:32801 (local address 127.18.229.190:33481)
0924 19:26:49.114370 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:49.114373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:49.114651 (+ 278us) server_negotiation.cc:407] Connection header received
0924 19:26:49.115970 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:49.115971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:49.115981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:49.116020 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:49.116845 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.117340 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.118452 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.118648 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.118736 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:49.121333 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:49.121335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:49.121335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:49.121342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:49.122845 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.123284 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.123285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.123285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.123335 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.123545 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.123546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.123546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.123651 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:49.123725 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:49.124022 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 19:26:49.124140 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:26:49.124430 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:49.114317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33481 (local address 127.18.229.130:32801)
0924 19:26:49.114506 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:49.114553 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:49.115860 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:49.116103 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:49.116132 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:49.116537 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:49.116742 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:49.116746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.117434 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.117435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:49.118348 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:49.118351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.118758 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.118759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:49.118899 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:49.119460 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:49.119471 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:49.121232 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:49.122920 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:49.122927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:49.122929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:49.123214 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:49.123394 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:49.123394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:49.123395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:49.123491 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:49.123733 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:49.123740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:49.123888 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:26:49.123999 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 19:26:49.124179 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260924 19:26:49.124897 23263 heartbeater.cc:347] Connected to a master server at 127.18.229.190:33481
I20260924 19:26:49.124979 23263 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
I20260924 19:26:49.125183 23263 heartbeater.cc:511] Master 127.18.229.190:33481 requested a full tablet report, sending...
I20260924 19:26:49.125582 22938 ts_manager.cc:194] Registered new tserver with Master: 50bded30aafb4c6c9a17295cffd47bb5 (127.18.229.130:35321)
I20260924 19:26:49.126010 22938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:32801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:49.139253 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33481
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:49.219156 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:49.219345 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:49.219414 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:49.220597 23268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:49.220635 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:49.220662 23268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:49.222119 23268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:49.222291 23268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:49.222458 23268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:49.227434 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:49.227456 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:49.227439 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:49.227699 23268 server_base.cc:1034] running on GCE node
I20260924 19:26:49.227941 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:49.228224 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:49.229362 23268 hybrid_clock.cc:648] HybridClock initialized: now 1790278009229350 us; error 28 us; skew 500 ppm
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:49.231740 23268 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:49.232221 23268 webserver.cc:460] Webserver started at http://127.18.229.131:39207/ using document root <none> and password file <none>
I20260924 19:26:49.232453 23268 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:49.232522 23268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:49.232663 23268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:49.233767 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "28762913730c452096676d6717f42c71"
format_stamp: "Formatted at 2026-09-24 19:26:49 on dist-test-slave-r8cc"
server_key: "7d55e557ade35b2f79c1d30a2f5b6fee"
server_key_iv: "936de3789b66b3ecacb4ec998c326b82"
server_key_version: "encryptionkey@0"
I20260924 19:26:49.234120 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "28762913730c452096676d6717f42c71"
format_stamp: "Formatted at 2026-09-24 19:26:49 on dist-test-slave-r8cc"
server_key: "7d55e557ade35b2f79c1d30a2f5b6fee"
server_key_iv: "936de3789b66b3ecacb4ec998c326b82"
server_key_version: "encryptionkey@0"
I20260924 19:26:49.235395 23268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:49.236112 23283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.236346 23268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:49.236431 23268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "28762913730c452096676d6717f42c71"
format_stamp: "Formatted at 2026-09-24 19:26:49 on dist-test-slave-r8cc"
server_key: "7d55e557ade35b2f79c1d30a2f5b6fee"
server_key_iv: "936de3789b66b3ecacb4ec998c326b82"
server_key_version: "encryptionkey@0"
I20260924 19:26:49.236514 23268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:49.259043 23268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:49.261480 23268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:49.261595 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:49.261777 23268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:49.262110 23268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:49.262146 23268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.262190 23268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:49.262217 23268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:49.268365 23268 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:34809
I20260924 19:26:49.268419 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:34809 every 8 connection(s)
I20260924 19:26:49.268734 23268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:49.274236 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23268
I20260924 19:26:49.274314 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:49.274499 19350 external_mini_cluster.cc:1428] Setting key 577fcf7d87c9710553ebf920057145c4
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:49.278004 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:49.269447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:36517 (local address 127.18.229.190:33481)
0924 19:26:49.269512 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:49.269514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:49.269699 (+ 185us) server_negotiation.cc:407] Connection header received
0924 19:26:49.270662 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:49.270663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:49.270677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:49.270728 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:49.271358 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.271861 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.272933 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.273136 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.273220 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:49.275421 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:49.275423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:49.275424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:49.275431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:49.276643 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.277061 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.277063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.277063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.277121 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.277368 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.277369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.277369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.277480 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:49.277558 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:49.277837 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:26:49.277913 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:26:49.278205 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:49.269449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33481 (local address 127.18.229.131:36517)
0924 19:26:49.269580 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:49.269615 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:49.270538 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:49.270809 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:49.270812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:49.271065 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:49.271255 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:49.271259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.271954 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.271956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:49.272828 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:49.272830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.273228 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.273229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:49.273352 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:49.273821 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:49.273829 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:49.275311 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:49.276738 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:49.276744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:49.276746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:49.276971 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:49.277192 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:49.277193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:49.277193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:49.277300 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:49.277548 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:49.277553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:49.277692 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 19:26:49.277815 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:26:49.277972 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260924 19:26:49.278662 23397 heartbeater.cc:347] Connected to a master server at 127.18.229.190:33481
I20260924 19:26:49.278779 23397 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:49.278966 23397 heartbeater.cc:511] Master 127.18.229.190:33481 requested a full tablet report, sending...
I20260924 19:26:49.279353 22938 ts_manager.cc:194] Registered new tserver with Master: 28762913730c452096676d6717f42c71 (127.18.229.131:34809)
I20260924 19:26:49.279944 22938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:36517
I20260924 19:26:49.286163 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:49.308848 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:49.301472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.18.229.129:38925)
0924 19:26:49.301678 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:49.301684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:49.301697 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:26:49.301730 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:49.301733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:49.301747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:49.301819 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:49.302395 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.303097 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.303801 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.304209 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.304268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:49.306415 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:49.306422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:49.306423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:49.306429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:49.307803 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.308095 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.308098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.308098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.308208 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.308396 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.308396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.308396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.308516 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:49.308625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:49.308676 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:26:49.308732 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[22893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:49.316644 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:49.310078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.18.229.190:33481)
0924 19:26:49.310127 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:49.310129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:49.310138 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:49.310202 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:49.310202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:49.310230 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:49.310322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:49.310766 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.311290 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.311972 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:49.312251 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:49.312352 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:49.314346 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:49.314347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:49.314347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:49.314355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:49.315470 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.315821 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.315822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.315823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.315881 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:49.316074 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:49.316076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:49.316076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:49.316252 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:49.316371 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:49.316471 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:26:49.316543 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 19:26:49.316896 22938 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40316
I20260924 19:26:49.317226 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23000
I20260924 19:26:49.322173 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23134
I20260924 19:26:49.326529 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23268
I20260924 19:26:49.330646 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 22906
2026-09-24T19:26:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4360 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[23409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[23409](info): set up 2 sockets
Sep 24 19:26:49 dist-test-slave-r8cc krb5kdc[23409](info): commencing operation
krb5kdc: starting...
W20260924 19:26:50.363994 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:50.479137 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44331
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:50.558832 23422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:50.559069 23422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:50.559109 23422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:50.560282 23422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:50.560354 23422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:50.560377 23422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:50.560398 23422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:50.560426 23422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:50.561877 23422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:44331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:50.562143 23422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:50.562361 23422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:50.567032 23427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:50.567070 23428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:50.567080 23430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:50.567211 23422 server_base.cc:1034] running on GCE node
I20260924 19:26:50.567487 23422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:50.567790 23422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:50.568950 23422 hybrid_clock.cc:648] HybridClock initialized: now 1790278010568935 us; error 45 us; skew 500 ppm
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:50.571264 23422 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:50.571626 23422 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:50.571696 23422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:50.571841 23422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:50.572944 23422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "4bea8c6bb60146b19146f5ea1307e93f"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "dc5c998938d12bc8a5248853158b33cc"
server_key_iv: "aa43e2a642094734f5e9cd21c0c0ab1c"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.573310 23422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "4bea8c6bb60146b19146f5ea1307e93f"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "dc5c998938d12bc8a5248853158b33cc"
server_key_iv: "aa43e2a642094734f5e9cd21c0c0ab1c"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.574600 23422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:26:50.575297 23436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.575467 23422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:50.575567 23422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "4bea8c6bb60146b19146f5ea1307e93f"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "dc5c998938d12bc8a5248853158b33cc"
server_key_iv: "aa43e2a642094734f5e9cd21c0c0ab1c"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.575649 23422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:50.592053 23422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:50.593986 23422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:50.594152 23422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:50.598124 23422 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:44331
I20260924 19:26:50.598152 23488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:44331 every 8 connection(s)
I20260924 19:26:50.598446 23422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:50.599363 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:50.601605 23489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f: Bootstrap starting.
I20260924 19:26:50.602138 23489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:50.602388 23489 log.cc:824] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:50.603016 23489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f: No bootstrap required, opened a new log
I20260924 19:26:50.603802 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23422
I20260924 19:26:50.603922 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:50.604092 19350 external_mini_cluster.cc:1428] Setting key f676b3a312fb01e28f0ea2793fa119e6
I20260924 19:26:50.604907 23489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bea8c6bb60146b19146f5ea1307e93f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44331 } }
I20260924 19:26:50.605068 23489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:50.605132 23489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea8c6bb60146b19146f5ea1307e93f, State: Initialized, Role: FOLLOWER
I20260924 19:26:50.605281 23489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bea8c6bb60146b19146f5ea1307e93f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44331 } }
I20260924 19:26:50.605386 23489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:50.605440 23489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:50.605504 23489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:50.606033 23489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bea8c6bb60146b19146f5ea1307e93f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44331 } }
I20260924 19:26:50.606153 23489 leader_election.cc:302] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bea8c6bb60146b19146f5ea1307e93f; no voters:
I20260924 19:26:50.606315 23489 leader_election.cc:288] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:50.606365 23494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:50.606470 23494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [term 1 LEADER]: Becoming Leader. State: Replica: 4bea8c6bb60146b19146f5ea1307e93f, State: Running, Role: LEADER
I20260924 19:26:50.606595 23489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:50.606619 23494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bea8c6bb60146b19146f5ea1307e93f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44331 } }
I20260924 19:26:50.606949 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea8c6bb60146b19146f5ea1307e93f. Latest consensus state: current_term: 1 leader_uuid: "4bea8c6bb60146b19146f5ea1307e93f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bea8c6bb60146b19146f5ea1307e93f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44331 } } }
I20260924 19:26:50.607057 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:50.606961 23496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bea8c6bb60146b19146f5ea1307e93f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bea8c6bb60146b19146f5ea1307e93f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44331 } } }
I20260924 19:26:50.607263 23496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:50.607609 23499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:50.608206 23499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:50.609771 23499 catalog_manager.cc:1347] Generated new cluster ID: e0a66382ae9e43b68865c71f06952722
I20260924 19:26:50.609875 23499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:50.615626 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:50.604681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.18.229.190:44331)
0924 19:26:50.604850 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:50.604857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:50.604882 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:26:50.604972 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:50.604977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:50.605017 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:50.605157 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:50.605893 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.607565 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.608753 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.609149 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.609284 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:50.611976 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:50.611980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:50.611981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:50.611993 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:50.613971 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:50.614348 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:50.614351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:50.614351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:50.614482 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:50.614672 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:50.614673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:50.614673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:50.614983 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:50.615154 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:50.615247 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 19:26:50.615386 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260924 19:26:50.625623 23499 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:50.626392 23499 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:50.632266 23499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f: Generated new TSK 0
I20260924 19:26:50.632462 23499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:50.661015 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44331
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:50.741518 23515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:50.741724 23515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:50.741747 23515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:50.742914 23515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:50.742956 23515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:50.742983 23515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:50.744510 23515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:50.744727 23515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:50.744980 23515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:50.749902 23521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:50.749890 23520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:50.750124 23515 server_base.cc:1034] running on GCE node
W20260924 19:26:50.749890 23523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:50.750418 23515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:50.750670 23515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:50.751816 23515 hybrid_clock.cc:648] HybridClock initialized: now 1790278010751795 us; error 35 us; skew 500 ppm
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:50.754477 23515 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:50.754853 23515 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:50.754901 23515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:50.755069 23515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:50.756095 23515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "9926f6a2443d4624b2053e556474b3b2"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "21a99299727e7acc45b989dd402b10af"
server_key_iv: "9093ad0c88a608ddee694203704a6c9b"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.756536 23515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "9926f6a2443d4624b2053e556474b3b2"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "21a99299727e7acc45b989dd402b10af"
server_key_iv: "9093ad0c88a608ddee694203704a6c9b"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.757776 23515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:50.758431 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.758598 23515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:50.758702 23515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "9926f6a2443d4624b2053e556474b3b2"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "21a99299727e7acc45b989dd402b10af"
server_key_iv: "9093ad0c88a608ddee694203704a6c9b"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.758775 23515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:50.765877 23515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:50.768066 23515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:50.768254 23515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:50.768469 23515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:50.768769 23515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:50.768822 23515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.768862 23515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:50.768890 23515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.774690 23515 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:42621
I20260924 19:26:50.774729 23642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:42621 every 8 connection(s)
I20260924 19:26:50.774976 23515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:50.776090 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23515
I20260924 19:26:50.776222 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:50.776401 19350 external_mini_cluster.cc:1428] Setting key 0b83b8b3585450e66f93a3f76a013a85
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
I20260924 19:26:50.784981 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:50.775652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:32905 (local address 127.18.229.190:44331)
0924 19:26:50.775717 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:50.775718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:50.775917 (+ 199us) server_negotiation.cc:407] Connection header received
0924 19:26:50.776649 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:50.776651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:50.776667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:50.776724 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:50.777951 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.778521 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.779694 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.779911 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.779996 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:50.782149 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:50.782151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:50.782152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:50.782160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:50.783523 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:50.783947 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:50.783948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:50.783949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:50.784007 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:50.784316 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:50.784317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:50.784317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:50.784442 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:50.784523 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:50.784801 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 19:26:50.784893 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:26:50.785140 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:50.775629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44331 (local address 127.18.229.129:32905)
0924 19:26:50.775771 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:50.775811 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:50.776514 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:50.776791 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:50.776799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:50.777146 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:50.777855 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:50.777862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.778614 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.778615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:50.779596 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:50.779598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.780006 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.780007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:50.780170 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:50.780710 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:50.780718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:50.782053 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:50.783624 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:50.783631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:50.783633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:50.783856 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:50.784077 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:50.784078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:50.784078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:50.784223 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:50.784513 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:50.784519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:50.784659 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:26:50.784760 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:26:50.784914 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260924 19:26:50.785657 23643 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44331
I20260924 19:26:50.785758 23643 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:50.785912 23643 heartbeater.cc:511] Master 127.18.229.190:44331 requested a full tablet report, sending...
I20260924 19:26:50.786422 23453 ts_manager.cc:194] Registered new tserver with Master: 9926f6a2443d4624b2053e556474b3b2 (127.18.229.129:42621)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
I20260924 19:26:50.787086 23453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:32905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:50.800197 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44331
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:50.882535 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:50.882756 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:50.882782 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:50.883934 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:50.883972 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:50.883998 23648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:50.885601 23648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:50.885818 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:50.886035 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:50.891182 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:50.891178 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:50.891357 23648 server_base.cc:1034] running on GCE node
W20260924 19:26:50.891182 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:50.891618 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:50.891884 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:50.893047 23648 hybrid_clock.cc:648] HybridClock initialized: now 1790278010893037 us; error 44 us; skew 500 ppm
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:50.895479 23648 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:50.895794 23648 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:50.895838 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:50.895927 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:50.896986 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "d1e7d21150984b64a2eb583a767d84d8"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "161bc9b4722c0b521f9e594956846c30"
server_key_iv: "55e8c6f36a9b9349f1996be72044fa82"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.897281 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "d1e7d21150984b64a2eb583a767d84d8"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "161bc9b4722c0b521f9e594956846c30"
server_key_iv: "55e8c6f36a9b9349f1996be72044fa82"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.898484 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:50.899165 23662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.899320 23648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:50.899406 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "d1e7d21150984b64a2eb583a767d84d8"
format_stamp: "Formatted at 2026-09-24 19:26:50 on dist-test-slave-r8cc"
server_key: "161bc9b4722c0b521f9e594956846c30"
server_key_iv: "55e8c6f36a9b9349f1996be72044fa82"
server_key_version: "encryptionkey@0"
I20260924 19:26:50.899464 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:50.916320 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:50.918651 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:50.918772 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:50.918955 23648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:50.919286 23648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:50.919322 23648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.919366 23648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:50.919397 23648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:50.925225 23648 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:46437
I20260924 19:26:50.925307 23775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:46437 every 8 connection(s)
I20260924 19:26:50.925573 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:26:50 dist-test-slave-r8cc krb5kdc[23409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:50.934293 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:50.926291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:39671 (local address 127.18.229.190:44331)
0924 19:26:50.926374 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:50.926377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:50.926542 (+ 165us) server_negotiation.cc:407] Connection header received
0924 19:26:50.926985 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:50.926986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:50.926997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:50.927049 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:50.927933 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.928456 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.929404 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.929585 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.929635 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:50.931698 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:50.931700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:50.931700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:50.931707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:50.933165 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:50.933525 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:50.933526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:50.933526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:50.933575 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:50.933785 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:50.933785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:50.933786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:50.933884 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:50.933933 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:50.934161 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 19:26:50.934225 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:26:50.934499 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:50.926244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44331 (local address 127.18.229.130:39671)
0924 19:26:50.926418 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:50.926454 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:50.926904 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:50.927102 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:50.927105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:50.927365 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:50.927859 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:50.927864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.928531 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.928532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:50.929329 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:50.929332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:50.929655 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:50.929656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:50.929754 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:50.930255 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:50.930271 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:50.931605 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:50.933233 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:50.933238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:50.933239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:50.933463 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:50.933630 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:50.933630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:50.933631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:50.933726 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:50.933937 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:50.933941 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:50.934055 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 19:26:50.934137 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:26:50.934251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260924 19:26:50.934971 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23648
I20260924 19:26:50.935078 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:50.935075 23776 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44331
I20260924 19:26:50.935178 23776 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:50.935281 19350 external_mini_cluster.cc:1428] Setting key 3c31e39e5806217835b473637cae461a
I20260924 19:26:50.935333 23776 heartbeater.cc:511] Master 127.18.229.190:44331 requested a full tablet report, sending...
I20260924 19:26:50.935877 23453 ts_manager.cc:194] Registered new tserver with Master: d1e7d21150984b64a2eb583a767d84d8 (127.18.229.130:46437)
I20260924 19:26:50.936336 23453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:39671
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:50.959648 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44331
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:51.039795 23781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:51.040009 23781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:51.040032 23781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:51.041227 23781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:51.041275 23781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:51.041301 23781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:51.042837 23781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:51.043100 23781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:51.043337 23781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:51.048295 23787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.048313 23786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.048295 23789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:51.048501 23781 server_base.cc:1034] running on GCE node
I20260924 19:26:51.048846 23781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:51.049132 23781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:51.050297 23781 hybrid_clock.cc:648] HybridClock initialized: now 1790278011050272 us; error 35 us; skew 500 ppm
Sep 24 19:26:51 dist-test-slave-r8cc krb5kdc[23409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:51.052772 23781 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:51.053112 23781 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:51.053181 23781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:51.053331 23781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:51.054423 23781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "2a747bd681e046c6b8da4e1e7066dba9"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "89abb443180584db53e79735c245695b"
server_key_iv: "859b3ba86025cf0d264e1f69ffe786f1"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.054778 23781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "2a747bd681e046c6b8da4e1e7066dba9"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "89abb443180584db53e79735c245695b"
server_key_iv: "859b3ba86025cf0d264e1f69ffe786f1"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.056011 23781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:51.056739 23795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.056905 23781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:51.057006 23781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "2a747bd681e046c6b8da4e1e7066dba9"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "89abb443180584db53e79735c245695b"
server_key_iv: "859b3ba86025cf0d264e1f69ffe786f1"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.057080 23781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:51.073665 23781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:51.075623 23781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:51.075786 23781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:51.076011 23781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:51.076419 23781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:51.076481 23781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.076524 23781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:51.076550 23781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.082423 23781 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:41155
I20260924 19:26:51.082458 23908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:41155 every 8 connection(s)
I20260924 19:26:51.082741 23781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:51.084609 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23781
I20260924 19:26:51.084721 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:51.084892 19350 external_mini_cluster.cc:1428] Setting key a3819e69322faef179cdbd1fe86f4371
Sep 24 19:26:51 dist-test-slave-r8cc krb5kdc[23409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:51.091871 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.083453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:51209 (local address 127.18.229.190:44331)
0924 19:26:51.083509 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.083511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.083724 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:26:51.084316 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.084318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.084331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.084393 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:51.085553 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.086137 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.087087 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.087267 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.087317 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.089366 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:51.089368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:51.089368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:51.089375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:51.090779 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:51.091107 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:51.091108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:51.091108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:51.091156 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:51.091363 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:51.091364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:51.091364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:51.091463 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:51.091513 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.091739 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.091805 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:26:51.092072 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.083429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44331 (local address 127.18.229.131:51209)
0924 19:26:51.083606 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.083640 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.084189 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.084476 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.084480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.084810 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:51.085474 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.085482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.086219 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.086219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.087018 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.087020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.087335 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.087336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.087433 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.087905 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:51.087913 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:51.089285 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:51.090846 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:51.090850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:51.090852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:51.091046 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:51.091211 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:51.091212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:51.091212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:51.091311 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:51.091518 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:51.091521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:51.091632 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.091713 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.091828 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260924 19:26:51.092614 23909 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44331
I20260924 19:26:51.092721 23909 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:51.092898 23909 heartbeater.cc:511] Master 127.18.229.190:44331 requested a full tablet report, sending...
I20260924 19:26:51.093266 23453 ts_manager.cc:194] Registered new tserver with Master: 2a747bd681e046c6b8da4e1e7066dba9 (127.18.229.131:41155)
I20260924 19:26:51.093600 23453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:51209
I20260924 19:26:51.096524 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:26:51.104305 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.097756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.18.229.190:44331)
0924 19:26:51.097813 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.097814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.097921 (+ 107us) server_negotiation.cc:407] Connection header received
0924 19:26:51.097958 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.097959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.097966 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.098033 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:51.098554 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.099006 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.099676 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.099850 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.099897 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.101277 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:51.101279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:51.101279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:51.101286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:51.102981 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:51.103449 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:51.103450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:51.103450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:51.103504 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:51.103707 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:51.103708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:51.103708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:51.103905 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:51.103983 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.104159 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.104234 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:26:51.105687 23453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:51.107363 23453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:51.113886 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.109848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41155 (local address 127.0.0.1:33876)
0924 19:26:51.110025 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.110036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.110088 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.110393 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.110395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.110397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:51.110488 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.110491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.111392 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.111393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.112591 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.112594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.113546 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.113547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.113640 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.113662 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.113751 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.113799 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260924 19:26:51.113905 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.110009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.18.229.131:41155)
0924 19:26:51.110181 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.110185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.110194 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:51.110257 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.110259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.110274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.110567 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:51.110643 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.111297 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.112703 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.113452 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.113545 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.113605 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.113789 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.113834 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260924 19:26:51.114054 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.109767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.18.229.130:46437)
0924 19:26:51.110206 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.110210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.110219 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:51.110252 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.110267 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.110285 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.110575 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:51.110656 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.111713 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.112946 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.113516 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.113848 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.113912 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.113939 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.113979 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":76,"threads_started":1}
I20260924 19:26:51.114151 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.109660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:46437 (local address 127.0.0.1:37962)
0924 19:26:51.109797 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.109839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.109907 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.110391 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.110394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.110396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:51.110490 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.110492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.111823 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.111824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.112859 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.112861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.113617 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.113618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.113689 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.113694 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.113722 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.113779 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":8,"thread_start_us":55,"threads_started":1}
I20260924 19:26:51.114609 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.109684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:42621 (local address 127.0.0.1:37010)
0924 19:26:51.110000 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.110009 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.110055 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.110449 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.110450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.110452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:51.110490 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.110493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.112196 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.112197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.113266 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.113269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.114389 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.114390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.114449 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.114453 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.114509 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.114547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":54,"threads_started":1}
I20260924 19:26:51.114638 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.109834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.18.229.129:42621)
0924 19:26:51.110178 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.110183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.110194 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:26:51.110237 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.110239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.110255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.110351 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:51.110573 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.111702 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.113357 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.114288 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.114390 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.114441 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.114524 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.114561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260924 19:26:51.115113 23843 tablet_service.cc:1467] Processing CreateTablet for tablet 851c2d1522394aa6979306147378d921 (DEFAULT_TABLE table=test-table [id=a2cd9074750f416faa5cefa75a439ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:51.115212 23577 tablet_service.cc:1467] Processing CreateTablet for tablet 851c2d1522394aa6979306147378d921 (DEFAULT_TABLE table=test-table [id=a2cd9074750f416faa5cefa75a439ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:51.115468 23843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851c2d1522394aa6979306147378d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:51.115471 23577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851c2d1522394aa6979306147378d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:51.116704 23710 tablet_service.cc:1467] Processing CreateTablet for tablet 851c2d1522394aa6979306147378d921 (DEFAULT_TABLE table=test-table [id=a2cd9074750f416faa5cefa75a439ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:51.117010 23710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851c2d1522394aa6979306147378d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:51.118145 23923 tablet_bootstrap.cc:492] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: Bootstrap starting.
I20260924 19:26:51.118626 23924 tablet_bootstrap.cc:492] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Bootstrap starting.
I20260924 19:26:51.118731 23923 tablet_bootstrap.cc:654] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:51.119293 23924 tablet_bootstrap.cc:654] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:51.119341 23923 log.cc:824] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:51.119832 23924 log.cc:824] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:51.120182 23923 tablet_bootstrap.cc:492] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: No bootstrap required, opened a new log
I20260924 19:26:51.120257 23923 ts_tablet_manager.cc:1397] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:51.120615 23924 tablet_bootstrap.cc:492] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: No bootstrap required, opened a new log
I20260924 19:26:51.120702 23924 ts_tablet_manager.cc:1397] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:51.121686 23923 raft_consensus.cc:348] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.121821 23923 raft_consensus.cc:374] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:51.121845 23923 raft_consensus.cc:729] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9926f6a2443d4624b2053e556474b3b2, State: Initialized, Role: FOLLOWER
I20260924 19:26:51.121968 23923 consensus_queue.cc:260] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.122118 23924 raft_consensus.cc:348] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.122260 23923 ts_tablet_manager.cc:1428] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:26:51.122282 23643 heartbeater.cc:503] Master 127.18.229.190:44331 was elected leader, sending a full tablet report...
I20260924 19:26:51.122298 23924 raft_consensus.cc:374] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:51.122342 23924 raft_consensus.cc:729] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a747bd681e046c6b8da4e1e7066dba9, State: Initialized, Role: FOLLOWER
I20260924 19:26:51.122437 23926 tablet_bootstrap.cc:492] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Bootstrap starting.
I20260924 19:26:51.122454 23924 consensus_queue.cc:260] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.122701 23924 ts_tablet_manager.cc:1428] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:51.122841 23909 heartbeater.cc:503] Master 127.18.229.190:44331 was elected leader, sending a full tablet report...
I20260924 19:26:51.123061 23926 tablet_bootstrap.cc:654] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:51.123631 23926 log.cc:824] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:51.124351 23926 tablet_bootstrap.cc:492] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: No bootstrap required, opened a new log
I20260924 19:26:51.124426 23926 ts_tablet_manager.cc:1397] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:51.125833 23926 raft_consensus.cc:348] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.125983 23926 raft_consensus.cc:374] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:51.126030 23926 raft_consensus.cc:729] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d21150984b64a2eb583a767d84d8, State: Initialized, Role: FOLLOWER
I20260924 19:26:51.126138 23926 consensus_queue.cc:260] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.126420 23926 ts_tablet_manager.cc:1428] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:26:51.126428 23776 heartbeater.cc:503] Master 127.18.229.190:44331 was elected leader, sending a full tablet report...
W20260924 19:26:51.176209 23777 tablet.cc:2369] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:26:51.275699 23644 tablet.cc:2369] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:26:51.333284 23910 tablet.cc:2369] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:51.485179 23931 raft_consensus.cc:482] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:51.485312 23931 raft_consensus.cc:504] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.485620 23931 leader_election.cc:288] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a747bd681e046c6b8da4e1e7066dba9 (127.18.229.131:41155), 9926f6a2443d4624b2053e556474b3b2 (127.18.229.129:42621)
I20260924 19:26:51.489734 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.485947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:47433 (local address 127.18.229.129:42621)
0924 19:26:51.485991 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.485993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.486025 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:26:51.486133 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.486136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.486147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.486226 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:51.486430 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.487187 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.488814 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.489373 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.489454 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.489462 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.489625 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.489664 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:51.489734 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.485916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:45895 (local address 127.18.229.131:41155)
0924 19:26:51.485965 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.485968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.486115 (+ 147us) server_negotiation.cc:407] Connection header received
0924 19:26:51.486159 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.486160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.486168 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.486226 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:51.486430 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.487187 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.488813 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.489383 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.489453 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.489462 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.489626 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.489663 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:51.489764 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.485833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:41155 (local address 127.18.229.130:45895)
0924 19:26:51.486042 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.486051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.486104 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.486250 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.486251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.486252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:51.486345 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.486348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.487284 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.487285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.488707 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.488711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.489463 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.489464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.489540 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.489545 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.489604 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.489654 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":80,"thread_start_us":57,"threads_started":1}
I20260924 19:26:51.489768 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.485833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:42621 (local address 127.18.229.130:47433)
0924 19:26:51.485960 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.485972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.486032 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.486244 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.486244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.486246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:51.486345 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.486348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.487275 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.487277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.488707 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:51.488711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.489461 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.489462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:51.489540 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.489546 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.489604 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.489654 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":70,"thread_start_us":59,"threads_started":1}
I20260924 19:26:51.490116 23863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851c2d1522394aa6979306147378d921" candidate_uuid: "d1e7d21150984b64a2eb583a767d84d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a747bd681e046c6b8da4e1e7066dba9" is_pre_election: true
I20260924 19:26:51.490300 23863 raft_consensus.cc:2454] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d21150984b64a2eb583a767d84d8 in term 0.
I20260924 19:26:51.490337 23597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851c2d1522394aa6979306147378d921" candidate_uuid: "d1e7d21150984b64a2eb583a767d84d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9926f6a2443d4624b2053e556474b3b2" is_pre_election: true
I20260924 19:26:51.490492 23597 raft_consensus.cc:2454] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d21150984b64a2eb583a767d84d8 in term 0.
I20260924 19:26:51.490541 23666 leader_election.cc:302] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a747bd681e046c6b8da4e1e7066dba9, d1e7d21150984b64a2eb583a767d84d8; no voters:
I20260924 19:26:51.490666 23931 raft_consensus.cc:2781] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:51.490736 23931 raft_consensus.cc:482] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:51.490764 23931 raft_consensus.cc:3037] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:51.491408 23931 raft_consensus.cc:504] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.491554 23931 leader_election.cc:288] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2a747bd681e046c6b8da4e1e7066dba9 (127.18.229.131:41155), 9926f6a2443d4624b2053e556474b3b2 (127.18.229.129:42621)
I20260924 19:26:51.491691 23863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851c2d1522394aa6979306147378d921" candidate_uuid: "d1e7d21150984b64a2eb583a767d84d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a747bd681e046c6b8da4e1e7066dba9"
I20260924 19:26:51.491711 23597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851c2d1522394aa6979306147378d921" candidate_uuid: "d1e7d21150984b64a2eb583a767d84d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9926f6a2443d4624b2053e556474b3b2"
I20260924 19:26:51.491768 23863 raft_consensus.cc:3037] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:51.491782 23597 raft_consensus.cc:3037] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:51.492712 23863 raft_consensus.cc:2454] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d21150984b64a2eb583a767d84d8 in term 1.
I20260924 19:26:51.492705 23597 raft_consensus.cc:2454] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d21150984b64a2eb583a767d84d8 in term 1.
I20260924 19:26:51.492887 23666 leader_election.cc:302] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a747bd681e046c6b8da4e1e7066dba9, d1e7d21150984b64a2eb583a767d84d8; no voters:
I20260924 19:26:51.492986 23931 raft_consensus.cc:2781] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:51.493162 23931 raft_consensus.cc:686] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 1 LEADER]: Becoming Leader. State: Replica: d1e7d21150984b64a2eb583a767d84d8, State: Running, Role: LEADER
I20260924 19:26:51.493285 23931 consensus_queue.cc:237] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } }
I20260924 19:26:51.494087 23453 catalog_manager.cc:5495] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e7d21150984b64a2eb583a767d84d8 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "d1e7d21150984b64a2eb583a767d84d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d21150984b64a2eb583a767d84d8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 46437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260924 19:26:51.558910 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.556821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.18.229.130:46437)
0924 19:26:51.556884 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.556886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.556897 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:26:51.556950 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.556951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.556968 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.557029 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:51.557156 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.557611 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.558298 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:51.558476 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:51.558524 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:51.558651 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:51.558746 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:51.558807 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.558837 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.558854 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:26:51.560861 23597 raft_consensus.cc:1261] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d21150984b64a2eb583a767d84d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:51.560882 23863 raft_consensus.cc:1261] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d21150984b64a2eb583a767d84d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:51.561249 23934 consensus_queue.cc:1035] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9926f6a2443d4624b2053e556474b3b2" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:51.561381 23931 consensus_queue.cc:1035] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a747bd681e046c6b8da4e1e7066dba9" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 41155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:51.563602 23937 mvcc.cc:205] Tried to move back new op lower bound from 7332978735350063104 to 7332978735077101568. Current Snapshot: MvccSnapshot[applied={T|T < 7332978735350063104}]
I20260924 19:26:51.564249 23939 mvcc.cc:205] Tried to move back new op lower bound from 7332978735350063104 to 7332978735077101568. Current Snapshot: MvccSnapshot[applied={T|T < 7332978735350063104}]
I20260924 19:26:51.564384 23938 mvcc.cc:205] Tried to move back new op lower bound from 7332978735350063104 to 7332978735077101568. Current Snapshot: MvccSnapshot[applied={T|T < 7332978735350063104}]
I20260924 19:26:51.565817 23451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:51.565905 23451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:51.566681 23451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bea8c6bb60146b19146f5ea1307e93f: Sending DeleteTablet for 3 replicas of tablet 851c2d1522394aa6979306147378d921
I20260924 19:26:51.567015 23710 tablet_service.cc:1514] Processing DeleteTablet for tablet 851c2d1522394aa6979306147378d921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:51 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:37962
I20260924 19:26:51.567071 23577 tablet_service.cc:1514] Processing DeleteTablet for tablet 851c2d1522394aa6979306147378d921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:51 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:37010
I20260924 19:26:51.567255 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23515
I20260924 19:26:51.567279 23843 tablet_service.cc:1514] Processing DeleteTablet for tablet 851c2d1522394aa6979306147378d921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:51 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:33876
I20260924 19:26:51.567390 23948 tablet_replica.cc:330] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2: stopping tablet replica
I20260924 19:26:51.567505 23948 raft_consensus.cc:2229] T 851c2d1522394aa6979306147378d921 P 9926f6a2443d4624b2053e556474b3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:51.569289 23947 tablet_replica.cc:330] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: stopping tablet replica
I20260924 19:26:51.569376 23947 raft_consensus.cc:2229] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:51.569446 23947 raft_consensus.cc:2258] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:51.569552 23949 tablet_replica.cc:330] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: stopping tablet replica
I20260924 19:26:51.569617 23949 raft_consensus.cc:2229] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:51.569684 23949 raft_consensus.cc:2258] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:51.570034 23949 ts_tablet_manager.cc:1905] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:51.570451 23947 ts_tablet_manager.cc:1905] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:51.571894 23949 ts_tablet_manager.cc:1918] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:51.571980 23949 log.cc:1196] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/851c2d1522394aa6979306147378d921
I20260924 19:26:51.572193 23947 ts_tablet_manager.cc:1918] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:51.572260 23949 ts_tablet_manager.cc:1939] T 851c2d1522394aa6979306147378d921 P 2a747bd681e046c6b8da4e1e7066dba9: Deleting consensus metadata
I20260924 19:26:51.572268 23947 log.cc:1196] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/851c2d1522394aa6979306147378d921
I20260924 19:26:51.572513 23947 ts_tablet_manager.cc:1939] T 851c2d1522394aa6979306147378d921 P d1e7d21150984b64a2eb583a767d84d8: Deleting consensus metadata
I20260924 19:26:51.572703 23440 catalog_manager.cc:4841] TS 2a747bd681e046c6b8da4e1e7066dba9 (127.18.229.131:41155): tablet 851c2d1522394aa6979306147378d921 (table test-table [id=a2cd9074750f416faa5cefa75a439ac8]) successfully deleted
I20260924 19:26:51.572996 23440 catalog_manager.cc:4841] TS d1e7d21150984b64a2eb583a767d84d8 (127.18.229.130:46437): tablet 851c2d1522394aa6979306147378d921 (table test-table [id=a2cd9074750f416faa5cefa75a439ac8]) successfully deleted
W20260924 19:26:51.573490 23438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:42621 (error 108)
I20260924 19:26:51.573777 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23648
I20260924 19:26:51.575178 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.575044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:42621 (local address 127.0.0.1:37024)
0924 19:26:51.575104 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.575141 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 19:26:51.575309 23438 catalog_manager.cc:4584] TS 9926f6a2443d4624b2053e556474b3b2 (127.18.229.129:42621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 851c2d1522394aa6979306147378d921: Network error: Client connection negotiation failed: client connection to 127.18.229.129:42621: connect: Connection refused (error 111)
I20260924 19:26:51.577932 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23781
I20260924 19:26:51.582373 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23422
2026-09-24T19:26:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2250 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T19:26:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:51Z Disabled control of system clock
I20260924 19:26:51.608852 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44763
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:44763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:26:51.687637 23954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:51.687852 23954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:51.687873 23954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:51.689085 23954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:51.689126 23954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:51.689139 23954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:51.689152 23954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:51.689195 23954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:51.690526 23954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:44763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:51.690790 23954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:51.691079 23954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:51.695843 23962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.695794 23959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.695789 23960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:51.696007 23954 server_base.cc:1034] running on GCE node
I20260924 19:26:51.696286 23954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:51.696557 23954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:51.697739 23954 hybrid_clock.cc:648] HybridClock initialized: now 1790278011697713 us; error 37 us; skew 500 ppm
I20260924 19:26:51.698892 23954 webserver.cc:460] Webserver started at http://127.18.229.190:37935/ using document root <none> and password file <none>
I20260924 19:26:51.699122 23954 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:51.699188 23954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:51.699345 23954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:51.700459 23954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "65249d12ec424ae9b733d470a6638e33"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "e85422aae8a123fbb52e184d382821bc"
server_key_iv: "d0785fd58d17e8533580ce6e7c58d0a8"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.700821 23954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "65249d12ec424ae9b733d470a6638e33"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "e85422aae8a123fbb52e184d382821bc"
server_key_iv: "d0785fd58d17e8533580ce6e7c58d0a8"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.702167 23954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:51.702924 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.703095 23954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:51.703233 23954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "65249d12ec424ae9b733d470a6638e33"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "e85422aae8a123fbb52e184d382821bc"
server_key_iv: "d0785fd58d17e8533580ce6e7c58d0a8"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.703313 23954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:51.706599 23954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:51.706861 23954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:51.707000 23954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:51.710623 23954 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:44763
I20260924 19:26:51.710668 24020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:44763 every 8 connection(s)
I20260924 19:26:51.711005 23954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:51.711606 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:51.712610 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 23954
I20260924 19:26:51.712738 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:51.712913 19350 external_mini_cluster.cc:1428] Setting key c27e0880c28b09d19f04326712020b96
I20260924 19:26:51.714965 24021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33: Bootstrap starting.
I20260924 19:26:51.715416 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.713408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.18.229.190:44763)
0924 19:26:51.713612 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.713619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.713642 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:26:51.714044 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.714050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.714092 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.714237 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:51.715023 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:51.715027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:51.715027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:51.715031 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:51.715079 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:51.715093 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:51.715147 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.715213 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.715267 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260924 19:26:51.715696 24021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:51.715983 24021 log.cc:824] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:51.716645 24021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33: No bootstrap required, opened a new log
I20260924 19:26:51.717929 24021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65249d12ec424ae9b733d470a6638e33" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44763 } }
I20260924 19:26:51.718052 24021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:51.718097 24021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65249d12ec424ae9b733d470a6638e33, State: Initialized, Role: FOLLOWER
I20260924 19:26:51.718219 24021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65249d12ec424ae9b733d470a6638e33" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44763 } }
I20260924 19:26:51.718290 24021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:51.718331 24021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:51.718379 24021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:51.719147 24021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65249d12ec424ae9b733d470a6638e33" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44763 } }
I20260924 19:26:51.719260 24021 leader_election.cc:302] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65249d12ec424ae9b733d470a6638e33; no voters:
I20260924 19:26:51.719414 24021 leader_election.cc:288] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:51.719444 24026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:51.719574 24026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [term 1 LEADER]: Becoming Leader. State: Replica: 65249d12ec424ae9b733d470a6638e33, State: Running, Role: LEADER
I20260924 19:26:51.719671 24026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65249d12ec424ae9b733d470a6638e33" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44763 } }
I20260924 19:26:51.719679 24021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:51.720018 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65249d12ec424ae9b733d470a6638e33. Latest consensus state: current_term: 1 leader_uuid: "65249d12ec424ae9b733d470a6638e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65249d12ec424ae9b733d470a6638e33" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44763 } } }
I20260924 19:26:51.720021 24027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65249d12ec424ae9b733d470a6638e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65249d12ec424ae9b733d470a6638e33" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44763 } } }
I20260924 19:26:51.720156 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:51.720163 24027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:51.720391 24033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:51.720918 24033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:51.722263 24033 catalog_manager.cc:1347] Generated new cluster ID: 3a2628d59df54d41a4f68142d725e11b
I20260924 19:26:51.722334 24033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:51.757499 24033 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:51.757701 24033 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:51.765821 24033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33: Generated new TSK 0
I20260924 19:26:51.766039 24033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:26:51.777637 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44763
--builtin_ntp_servers=127.18.229.148:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:26:51.861325 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:51.861519 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:51.861542 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:51.862699 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:51.862738 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:51.862797 24045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:51.864190 24045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:51.864431 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:51.864720 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:51.869761 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.869772 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:51.869804 24045 server_base.cc:1034] running on GCE node
W20260924 19:26:51.869773 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:51.870160 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:51.870369 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:51.871505 24045 hybrid_clock.cc:648] HybridClock initialized: now 1790278011871493 us; error 29 us; skew 500 ppm
I20260924 19:26:51.872751 24045 webserver.cc:460] Webserver started at http://127.18.229.129:44979/ using document root <none> and password file <none>
I20260924 19:26:51.873001 24045 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:51.873045 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:51.873210 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:51.874284 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "9e1dd72db21e4654aa04228bfe53faca"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "d33815e87d5d9564768f58e9081fabd2"
server_key_iv: "10baa9fde5044e71520935a64e5f59c9"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.874631 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "9e1dd72db21e4654aa04228bfe53faca"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "d33815e87d5d9564768f58e9081fabd2"
server_key_iv: "10baa9fde5044e71520935a64e5f59c9"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.875901 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:51.876644 24059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.876822 24045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.876911 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "9e1dd72db21e4654aa04228bfe53faca"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "d33815e87d5d9564768f58e9081fabd2"
server_key_iv: "10baa9fde5044e71520935a64e5f59c9"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.876994 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:51.880262 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:51.880525 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:51.880664 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:51.880870 24045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:51.881165 24045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:51.881217 24045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.881263 24045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:51.881289 24045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.886969 24045 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:45771
I20260924 19:26:51.887003 24172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:45771 every 8 connection(s)
I20260924 19:26:51.887343 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:51.890429 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.888087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:47523 (local address 127.18.229.190:44763)
0924 19:26:51.888172 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:51.888174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:51.888419 (+ 245us) server_negotiation.cc:407] Connection header received
0924 19:26:51.888919 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:51.888920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:51.888928 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:51.888996 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:51.890164 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:51.890165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:51.890166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:51.890170 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:51.890185 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:51.890190 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:51.890242 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:51.890338 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:26:51.890353 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:26:51.890627 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:51.888089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44763 (local address 127.18.229.129:47523)
0924 19:26:51.888298 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:51.888331 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:51.888811 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:51.889071 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:51.889075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:51.889083 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:51.890011 (+ 928us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:51.890012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:51.890040 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:51.890067 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:51.890239 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:51.890243 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:51.890246 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:51.890349 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:26:51.890409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260924 19:26:51.891137 24173 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44763
I20260924 19:26:51.891227 24173 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:51.891340 24173 heartbeater.cc:511] Master 127.18.229.190:44763 requested a full tablet report, sending...
I20260924 19:26:51.891839 23985 ts_manager.cc:194] Registered new tserver with Master: 9e1dd72db21e4654aa04228bfe53faca (127.18.229.129:45771)
I20260924 19:26:51.892405 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24045
I20260924 19:26:51.892504 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:51.892684 19350 external_mini_cluster.cc:1428] Setting key f9123fc25777bf4e5ca572c3223581f8
I20260924 19:26:51.893426 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44763
--builtin_ntp_servers=127.18.229.148:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:26:51.973462 24176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:51.973704 24176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:51.973737 24176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:51.974836 24176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:51.974897 24176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:51.974937 24176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:51.976336 24176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:51.976601 24176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:51.976884 24176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:51.981848 24184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.981843 24182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:51.981890 24181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:51.982141 24176 server_base.cc:1034] running on GCE node
I20260924 19:26:51.982368 24176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:51.982599 24176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:51.983770 24176 hybrid_clock.cc:648] HybridClock initialized: now 1790278011983759 us; error 28 us; skew 500 ppm
I20260924 19:26:51.984988 24176 webserver.cc:460] Webserver started at http://127.18.229.130:33279/ using document root <none> and password file <none>
I20260924 19:26:51.985225 24176 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:51.985293 24176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:51.985422 24176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:51.986431 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "ce96dcb5048a4a80b72c4b61957a9110"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "66beca5ba9249426a75a88375a2eccf7"
server_key_iv: "f4a923e1499812eaa0909eeb0e055126"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.986814 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "ce96dcb5048a4a80b72c4b61957a9110"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "66beca5ba9249426a75a88375a2eccf7"
server_key_iv: "f4a923e1499812eaa0909eeb0e055126"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.988168 24176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:51.988823 24190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.988970 24176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:51.989064 24176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "ce96dcb5048a4a80b72c4b61957a9110"
format_stamp: "Formatted at 2026-09-24 19:26:51 on dist-test-slave-r8cc"
server_key: "66beca5ba9249426a75a88375a2eccf7"
server_key_iv: "f4a923e1499812eaa0909eeb0e055126"
server_key_version: "encryptionkey@0"
I20260924 19:26:51.989142 24176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:51.992345 24176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:51.992583 24176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:51.992756 24176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:51.992952 24176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:51.993242 24176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:51.993300 24176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.993340 24176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:51.993366 24176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:51.998997 24176 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:43179
I20260924 19:26:51.999029 24303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:43179 every 8 connection(s)
I20260924 19:26:51.999331 24176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:52.002063 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.000037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:60955 (local address 127.18.229.190:44763)
0924 19:26:52.000223 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.000225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.000310 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:26:52.000739 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.000740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.000745 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.000804 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.001882 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.001883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.001883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.001887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.001900 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.001904 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.001948 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.001999 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.002010 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 19:26:52.002223 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.000009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44763 (local address 127.18.229.130:60955)
0924 19:26:52.000192 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.000229 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.000651 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.000852 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.000855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.000863 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.001743 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.001743 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.001771 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.001796 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.001953 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.001955 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.001957 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.002005 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.002048 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 19:26:52.002731 24304 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44763
I20260924 19:26:52.002805 24304 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:52.002961 24304 heartbeater.cc:511] Master 127.18.229.190:44763 requested a full tablet report, sending...
I20260924 19:26:52.003311 23985 ts_manager.cc:194] Registered new tserver with Master: ce96dcb5048a4a80b72c4b61957a9110 (127.18.229.130:43179)
I20260924 19:26:52.007814 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24176
I20260924 19:26:52.007890 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:52.008019 19350 external_mini_cluster.cc:1428] Setting key 4c94e071830ebe0c8d70a21d7004e6dd
I20260924 19:26:52.008877 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44763
--builtin_ntp_servers=127.18.229.148:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:26:52.087910 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:52.088125 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:52.088150 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:52.089326 24307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:52.089366 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:52.089392 24307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:52.090728 24307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:52.090962 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:52.091239 24307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:52.096529 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:52.096550 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:52.096550 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.096784 24307 server_base.cc:1034] running on GCE node
I20260924 19:26:52.097005 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:52.097270 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:52.098425 24307 hybrid_clock.cc:648] HybridClock initialized: now 1790278012098407 us; error 29 us; skew 500 ppm
I20260924 19:26:52.099711 24307 webserver.cc:460] Webserver started at http://127.18.229.131:44115/ using document root <none> and password file <none>
I20260924 19:26:52.099957 24307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:52.100032 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:52.100183 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:52.101224 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "17cce734c76e44fab66d802dd77a24ce"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "2c22df92956fa9e0803bfd28ed778ffd"
server_key_iv: "d6201db4f8b9993b88bb9b595fe7f093"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.101588 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "17cce734c76e44fab66d802dd77a24ce"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "2c22df92956fa9e0803bfd28ed778ffd"
server_key_iv: "d6201db4f8b9993b88bb9b595fe7f093"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.102856 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:52.103525 24321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.103686 24307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:52.103786 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "17cce734c76e44fab66d802dd77a24ce"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "2c22df92956fa9e0803bfd28ed778ffd"
server_key_iv: "d6201db4f8b9993b88bb9b595fe7f093"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.103868 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:52.108309 24307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:52.108556 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:52.108696 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:52.108906 24307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:52.109210 24307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:52.109265 24307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.109303 24307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:52.109331 24307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.114748 24307 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:43981
I20260924 19:26:52.114784 24434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:43981 every 8 connection(s)
I20260924 19:26:52.115077 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:52.117831 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.115747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:60895 (local address 127.18.229.190:44763)
0924 19:26:52.115810 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.115812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.116008 (+ 196us) server_negotiation.cc:407] Connection header received
0924 19:26:52.116492 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.116493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.116498 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.116560 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.117663 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.117664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.117664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.117666 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.117676 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.117679 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.117717 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.117770 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.117781 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:26:52.118019 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.115723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44763 (local address 127.18.229.131:60895)
0924 19:26:52.115888 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.115926 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.116404 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.116613 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.116616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.116624 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.117541 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.117542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.117568 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.117594 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.117719 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.117721 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.117723 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.117778 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.117822 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260924 19:26:52.118538 24435 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44763
I20260924 19:26:52.118610 24435 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:52.118718 24435 heartbeater.cc:511] Master 127.18.229.190:44763 requested a full tablet report, sending...
I20260924 19:26:52.119060 23985 ts_manager.cc:194] Registered new tserver with Master: 17cce734c76e44fab66d802dd77a24ce (127.18.229.131:43981)
I20260924 19:26:52.123354 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24307
I20260924 19:26:52.123435 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:52.123569 19350 external_mini_cluster.cc:1428] Setting key 0608f5b8bf4583caaa11d702c75da5d7
I20260924 19:26:52.124380 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:26:52.126499 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.125592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.18.229.190:44763)
0924 19:26:52.125626 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.125628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.125728 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:26:52.125776 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.125777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.125780 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.125831 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.126368 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.126369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.126369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.126371 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.126379 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.126382 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.126418 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.126449 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.126458 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:26:52.127904 23985 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:52.129485 23985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:52.133112 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.131730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.18.229.129:45771)
0924 19:26:52.131899 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.131904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.131914 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:52.131940 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.131943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.131952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.132061 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.132759 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.132766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.132766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.132771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.132805 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.132828 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.132983 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.133006 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.133021 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260924 19:26:52.133275 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.131525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.18.229.130:43179)
0924 19:26:52.131686 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.131691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.131700 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:52.131723 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.131725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.131733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.131825 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.132555 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.132562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.132562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.132566 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.132627 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.132651 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.132721 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.133167 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.133184 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260924 19:26:52.132905 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.131358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:43179 (local address 127.0.0.1:43616)
0924 19:26:52.131543 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.131582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.131634 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.131825 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.131830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.131836 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.132451 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.132451 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.132466 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.132488 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.132732 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.132739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.132742 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.132802 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.132815 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":53,"threads_started":1}
I20260924 19:26:52.133487 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.131386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45771 (local address 127.0.0.1:46524)
0924 19:26:52.131714 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.131725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.131784 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.132235 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.132236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.132238 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.132643 (+ 405us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.132643 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.132651 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.132665 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.132899 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.132900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.132900 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.132934 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.132945 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":87,"threads_started":1}
I20260924 19:26:52.133622 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.131589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.18.229.131:43981)
0924 19:26:52.131916 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.131920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.132290 (+ 370us) server_negotiation.cc:407] Connection header received
0924 19:26:52.132706 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.132711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.132720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.132798 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.133281 (+ 483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.133286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.133287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.133291 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.133321 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.133339 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.133388 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.133517 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.133531 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":60,"threads_started":1}
I20260924 19:26:52.133819 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.131404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:43981 (local address 127.0.0.1:50756)
0924 19:26:52.131963 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.131975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.132318 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.132800 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.132801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.132804 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.133152 (+ 348us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.133153 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.133158 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.133185 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.133412 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.133413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.133413 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.133456 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.133470 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":83,"threads_started":1}
I20260924 19:26:52.134115 24238 tablet_service.cc:1467] Processing CreateTablet for tablet b55af99bcf0a4fb1869e4d91cfeb17b7 (DEFAULT_TABLE table=test-table [id=8484b6af071a414ab30ec4a72b3bdd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:52.134176 24107 tablet_service.cc:1467] Processing CreateTablet for tablet b55af99bcf0a4fb1869e4d91cfeb17b7 (DEFAULT_TABLE table=test-table [id=8484b6af071a414ab30ec4a72b3bdd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:52.134497 24238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55af99bcf0a4fb1869e4d91cfeb17b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:52.134503 24107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55af99bcf0a4fb1869e4d91cfeb17b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:52.136797 24369 tablet_service.cc:1467] Processing CreateTablet for tablet b55af99bcf0a4fb1869e4d91cfeb17b7 (DEFAULT_TABLE table=test-table [id=8484b6af071a414ab30ec4a72b3bdd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:52.137120 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55af99bcf0a4fb1869e4d91cfeb17b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:52.137132 24449 tablet_bootstrap.cc:492] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: Bootstrap starting.
I20260924 19:26:52.137408 24450 tablet_bootstrap.cc:492] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Bootstrap starting.
I20260924 19:26:52.137876 24449 tablet_bootstrap.cc:654] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:52.138028 24450 tablet_bootstrap.cc:654] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:52.138221 24449 log.cc:824] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:52.138340 24450 log.cc:824] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:52.138989 24450 tablet_bootstrap.cc:492] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: No bootstrap required, opened a new log
I20260924 19:26:52.139014 24449 tablet_bootstrap.cc:492] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: No bootstrap required, opened a new log
I20260924 19:26:52.139081 24449 ts_tablet_manager.cc:1397] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:52.139082 24450 ts_tablet_manager.cc:1397] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:52.140453 24453 tablet_bootstrap.cc:492] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Bootstrap starting.
I20260924 19:26:52.141139 24453 tablet_bootstrap.cc:654] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:52.141481 24453 log.cc:824] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:52.141467 24449 raft_consensus.cc:348] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.141630 24449 raft_consensus.cc:374] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:52.141670 24449 raft_consensus.cc:729] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1dd72db21e4654aa04228bfe53faca, State: Initialized, Role: FOLLOWER
I20260924 19:26:52.141654 24450 raft_consensus.cc:348] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.141811 24450 raft_consensus.cc:374] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:52.141780 24449 consensus_queue.cc:260] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.141850 24450 raft_consensus.cc:729] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce96dcb5048a4a80b72c4b61957a9110, State: Initialized, Role: FOLLOWER
I20260924 19:26:52.141959 24450 consensus_queue.cc:260] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.142056 24173 heartbeater.cc:503] Master 127.18.229.190:44763 was elected leader, sending a full tablet report...
I20260924 19:26:52.142197 24450 ts_tablet_manager.cc:1428] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:52.142215 24304 heartbeater.cc:503] Master 127.18.229.190:44763 was elected leader, sending a full tablet report...
I20260924 19:26:52.142052 24449 ts_tablet_manager.cc:1428] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:52.142915 24453 tablet_bootstrap.cc:492] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: No bootstrap required, opened a new log
I20260924 19:26:52.142973 24453 ts_tablet_manager.cc:1397] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:26:52.144856 24453 raft_consensus.cc:348] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.144987 24453 raft_consensus.cc:374] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:52.145031 24453 raft_consensus.cc:729] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cce734c76e44fab66d802dd77a24ce, State: Initialized, Role: FOLLOWER
I20260924 19:26:52.145133 24453 consensus_queue.cc:260] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.145362 24453 ts_tablet_manager.cc:1428] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:26:52.145426 24435 heartbeater.cc:503] Master 127.18.229.190:44763 was elected leader, sending a full tablet report...
I20260924 19:26:52.233460 24456 raft_consensus.cc:482] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:52.233577 24456 raft_consensus.cc:504] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.233973 24456 leader_election.cc:288] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e1dd72db21e4654aa04228bfe53faca (127.18.229.129:45771), 17cce734c76e44fab66d802dd77a24ce (127.18.229.131:43981)
I20260924 19:26:52.235391 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.234167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45771 (local address 127.18.229.130:44319)
0924 19:26:52.234223 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.234236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.234295 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.234449 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.234451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.234453 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.234972 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.234973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.234981 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.234995 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.235238 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.235239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.235240 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.235305 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.235320 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:26:52.235435 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.234273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:44319 (local address 127.18.229.129:45771)
0924 19:26:52.234328 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.234330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.234338 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:52.234364 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.234365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.234370 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.234444 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.235101 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.235103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.235103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.235122 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.235144 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.235150 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.235215 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.235318 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.235349 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:26:52.235498 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.234168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:43981 (local address 127.18.229.130:59333)
0924 19:26:52.234398 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.234405 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.234476 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.234618 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.234619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.234621 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:52.235160 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:26:52.235161 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:52.235167 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:26:52.235179 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:52.235356 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:52.235357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:52.235357 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 19:26:52.235409 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:26:52.235420 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":53,"thread_start_us":101,"threads_started":1}
I20260924 19:26:52.235495 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.234290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:59333 (local address 127.18.229.131:43981)
0924 19:26:52.234341 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.234344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.234472 (+ 128us) server_negotiation.cc:407] Connection header received
0924 19:26:52.234534 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.234535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.234541 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.234614 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.235261 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.235262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.235262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.235267 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.235285 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.235290 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.235346 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.235406 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.235420 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:26:52.235715 24127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55af99bcf0a4fb1869e4d91cfeb17b7" candidate_uuid: "ce96dcb5048a4a80b72c4b61957a9110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1dd72db21e4654aa04228bfe53faca" is_pre_election: true
I20260924 19:26:52.235858 24127 raft_consensus.cc:2454] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce96dcb5048a4a80b72c4b61957a9110 in term 0.
I20260924 19:26:52.235836 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55af99bcf0a4fb1869e4d91cfeb17b7" candidate_uuid: "ce96dcb5048a4a80b72c4b61957a9110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cce734c76e44fab66d802dd77a24ce" is_pre_election: true
I20260924 19:26:52.235988 24389 raft_consensus.cc:2454] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce96dcb5048a4a80b72c4b61957a9110 in term 0.
I20260924 19:26:52.236142 24192 leader_election.cc:302] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1dd72db21e4654aa04228bfe53faca, ce96dcb5048a4a80b72c4b61957a9110; no voters:
I20260924 19:26:52.236292 24456 raft_consensus.cc:2781] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:52.236358 24456 raft_consensus.cc:482] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:52.236375 24456 raft_consensus.cc:3037] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:52.237020 24456 raft_consensus.cc:504] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.237154 24456 leader_election.cc:288] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [CANDIDATE]: Term 1 election: Requested vote from peers 9e1dd72db21e4654aa04228bfe53faca (127.18.229.129:45771), 17cce734c76e44fab66d802dd77a24ce (127.18.229.131:43981)
I20260924 19:26:52.237349 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55af99bcf0a4fb1869e4d91cfeb17b7" candidate_uuid: "ce96dcb5048a4a80b72c4b61957a9110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17cce734c76e44fab66d802dd77a24ce"
I20260924 19:26:52.237353 24127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55af99bcf0a4fb1869e4d91cfeb17b7" candidate_uuid: "ce96dcb5048a4a80b72c4b61957a9110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1dd72db21e4654aa04228bfe53faca"
I20260924 19:26:52.237438 24389 raft_consensus.cc:3037] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:52.237438 24127 raft_consensus.cc:3037] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:52.238356 24127 raft_consensus.cc:2454] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce96dcb5048a4a80b72c4b61957a9110 in term 1.
I20260924 19:26:52.238356 24389 raft_consensus.cc:2454] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce96dcb5048a4a80b72c4b61957a9110 in term 1.
I20260924 19:26:52.238517 24192 leader_election.cc:302] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1dd72db21e4654aa04228bfe53faca, ce96dcb5048a4a80b72c4b61957a9110; no voters:
I20260924 19:26:52.238608 24456 raft_consensus.cc:2781] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:52.238773 24456 raft_consensus.cc:686] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 1 LEADER]: Becoming Leader. State: Replica: ce96dcb5048a4a80b72c4b61957a9110, State: Running, Role: LEADER
I20260924 19:26:52.238894 24456 consensus_queue.cc:237] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } }
I20260924 19:26:52.239691 23985 catalog_manager.cc:5495] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce96dcb5048a4a80b72c4b61957a9110 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "ce96dcb5048a4a80b72c4b61957a9110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce96dcb5048a4a80b72c4b61957a9110" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 43179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 } health_report { overall_health: UNKNOWN } } }
I20260924 19:26:52.248101 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.247145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.18.229.130:43179)
0924 19:26:52.247186 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.247188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.247196 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:52.247231 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.247236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.247248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.247296 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.247878 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.247880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.247880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.247883 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.247896 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.247900 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.247951 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.247984 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.247995 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:26:52.249800 24127 raft_consensus.cc:1261] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 1 FOLLOWER]: Refusing update from remote peer ce96dcb5048a4a80b72c4b61957a9110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:52.249804 24389 raft_consensus.cc:1261] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 1 FOLLOWER]: Refusing update from remote peer ce96dcb5048a4a80b72c4b61957a9110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 19:26:52.249836 24305 tablet.cc:2369] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:52.250145 24459 consensus_queue.cc:1035] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17cce734c76e44fab66d802dd77a24ce" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:52.250296 24456 consensus_queue.cc:1035] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1dd72db21e4654aa04228bfe53faca" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:52.252722 24462 mvcc.cc:205] Tried to move back new op lower bound from 7332978738172497920 to 7332978738131283968. Current Snapshot: MvccSnapshot[applied={T|T < 7332978738172497920}]
I20260924 19:26:52.253147 24463 mvcc.cc:205] Tried to move back new op lower bound from 7332978738172497920 to 7332978738131283968. Current Snapshot: MvccSnapshot[applied={T|T < 7332978738172497920}]
I20260924 19:26:52.254945 23984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:52.255062 23984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:52.255816 23984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65249d12ec424ae9b733d470a6638e33: Sending DeleteTablet for 3 replicas of tablet b55af99bcf0a4fb1869e4d91cfeb17b7
I20260924 19:26:52.256191 24238 tablet_service.cc:1514] Processing DeleteTablet for tablet b55af99bcf0a4fb1869e4d91cfeb17b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:52 UTC) from {username='slave'} at 127.0.0.1:43616
I20260924 19:26:52.256210 24369 tablet_service.cc:1514] Processing DeleteTablet for tablet b55af99bcf0a4fb1869e4d91cfeb17b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:52 UTC) from {username='slave'} at 127.0.0.1:50756
I20260924 19:26:52.256258 24107 tablet_service.cc:1514] Processing DeleteTablet for tablet b55af99bcf0a4fb1869e4d91cfeb17b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:52 UTC) from {username='slave'} at 127.0.0.1:46524
I20260924 19:26:52.256430 24472 tablet_replica.cc:330] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: stopping tablet replica
I20260924 19:26:52.256482 24473 tablet_replica.cc:330] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca: stopping tablet replica
I20260924 19:26:52.256524 24472 raft_consensus.cc:2229] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:52.256551 24473 raft_consensus.cc:2229] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:52.256618 24472 raft_consensus.cc:2258] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:52.256624 24473 raft_consensus.cc:2258] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 9e1dd72db21e4654aa04228bfe53faca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:52.256793 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24045
I20260924 19:26:52.257038 24472 ts_tablet_manager.cc:1905] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:52.258555 24472 ts_tablet_manager.cc:1918] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:52.258653 24472 log.cc:1196] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/b55af99bcf0a4fb1869e4d91cfeb17b7
I20260924 19:26:52.258904 24472 ts_tablet_manager.cc:1939] T b55af99bcf0a4fb1869e4d91cfeb17b7 P 17cce734c76e44fab66d802dd77a24ce: Deleting consensus metadata
I20260924 19:26:52.259227 23971 catalog_manager.cc:4841] TS 17cce734c76e44fab66d802dd77a24ce (127.18.229.131:43981): tablet b55af99bcf0a4fb1869e4d91cfeb17b7 (table test-table [id=8484b6af071a414ab30ec4a72b3bdd54]) successfully deleted
W20260924 19:26:52.263432 23970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:45771 (error 108)
I20260924 19:26:52.263473 24471 tablet_replica.cc:330] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: stopping tablet replica
I20260924 19:26:52.263562 24471 raft_consensus.cc:2229] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:52.263675 24471 raft_consensus.cc:2258] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:52.263959 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24176
I20260924 19:26:52.264082 24471 ts_tablet_manager.cc:1905] T b55af99bcf0a4fb1869e4d91cfeb17b7 P ce96dcb5048a4a80b72c4b61957a9110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:52.264626 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.263880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45771 (local address 127.0.0.1:46526)
0924 19:26:52.264542 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.264597 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
W20260924 19:26:52.264755 23970 catalog_manager.cc:4584] TS 9e1dd72db21e4654aa04228bfe53faca (127.18.229.129:45771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b55af99bcf0a4fb1869e4d91cfeb17b7: Network error: Client connection negotiation failed: client connection to 127.18.229.129:45771: connect: Connection refused (error 111)
I20260924 19:26:52.268949 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24307
I20260924 19:26:52.269029 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.268901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:43179 (local address 127.0.0.1:43622)
0924 19:26:52.268944 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.268969 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:43179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 19:26:52.273603 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 23954
2026-09-24T19:26:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (690 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 19:26:52.292968 19350 mini_oidc.cc:152] Starting JWKS server
I20260924 19:26:52.294103 19350 webserver.cc:460] Webserver started at https://127.0.0.1:38261/ using document root <none> and password file <none>
I20260924 19:26:52.294229 19350 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:26:52.294418 19350 webserver.cc:460] Webserver started at http://127.0.0.1:43365/ using document root <none> and password file <none>
2026-09-24T19:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:52Z Disabled control of system clock
I20260924 19:26:52.301851 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:46257
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:46257
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38261/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/testchainca.cert with env {}
W20260924 19:26:52.379555 24480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:52.379765 24480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:52.379787 24480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:52.380923 24480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:52.380955 24480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:52.380968 24480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:52.380980 24480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:52.380992 24480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:26:52.381004 24480 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:26:52.381016 24480 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38261/jwks.json
I20260924 19:26:52.382478 24480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:46257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38261/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:52.382665 24480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:52.382817 24480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:52.387544 24486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.387665 24480 server_base.cc:1034] running on GCE node
W20260924 19:26:52.387560 24485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:52.387557 24488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.387972 24480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:52.388309 24480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:52.389460 24480 hybrid_clock.cc:648] HybridClock initialized: now 1790278012389447 us; error 29 us; skew 500 ppm
I20260924 19:26:52.390609 24480 webserver.cc:460] Webserver started at http://127.18.229.190:33639/ using document root <none> and password file <none>
I20260924 19:26:52.390842 24480 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:52.390888 24480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:52.391036 24480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:52.392095 24480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "db38f22395da4f9baa8c7c82d607c7f4"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "6b151d3769ecf5519fda3264a23dd9ad"
server_key_iv: "992f5bf09fd286813d632cd1d2bddfa8"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.392529 24480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "db38f22395da4f9baa8c7c82d607c7f4"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "6b151d3769ecf5519fda3264a23dd9ad"
server_key_iv: "992f5bf09fd286813d632cd1d2bddfa8"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.393782 24480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:52.394483 24494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.394627 24480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.394729 24480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "db38f22395da4f9baa8c7c82d607c7f4"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "6b151d3769ecf5519fda3264a23dd9ad"
server_key_iv: "992f5bf09fd286813d632cd1d2bddfa8"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.394804 24480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:52.427222 24480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:52.427570 24480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:52.427726 24480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:52.431898 24480 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:46257
I20260924 19:26:52.431937 24546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:46257 every 8 connection(s)
I20260924 19:26:52.432310 24480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:52.433408 24547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:52.435644 24547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4: Bootstrap starting.
I20260924 19:26:52.436259 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24480
I20260924 19:26:52.436305 24547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:52.436371 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:52.436616 19350 external_mini_cluster.cc:1428] Setting key 413f371d43c6df7bb5f0184e8817f387
I20260924 19:26:52.436637 24547 log.cc:824] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:52.437587 24547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4: No bootstrap required, opened a new log
I20260924 19:26:52.439694 24547 raft_consensus.cc:348] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38f22395da4f9baa8c7c82d607c7f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46257 } }
I20260924 19:26:52.439859 24547 raft_consensus.cc:374] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:52.439889 24547 raft_consensus.cc:729] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db38f22395da4f9baa8c7c82d607c7f4, State: Initialized, Role: FOLLOWER
I20260924 19:26:52.440001 24547 consensus_queue.cc:260] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38f22395da4f9baa8c7c82d607c7f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46257 } }
I20260924 19:26:52.440069 24547 raft_consensus.cc:388] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:52.440097 24547 raft_consensus.cc:482] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:52.440188 24547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:52.440749 24547 raft_consensus.cc:504] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38f22395da4f9baa8c7c82d607c7f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46257 } }
I20260924 19:26:52.440858 24547 leader_election.cc:302] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db38f22395da4f9baa8c7c82d607c7f4; no voters:
I20260924 19:26:52.441026 24547 leader_election.cc:288] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:52.441077 24552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:52.441210 24552 raft_consensus.cc:686] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [term 1 LEADER]: Becoming Leader. State: Replica: db38f22395da4f9baa8c7c82d607c7f4, State: Running, Role: LEADER
I20260924 19:26:52.441309 24552 consensus_queue.cc:237] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38f22395da4f9baa8c7c82d607c7f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46257 } }
I20260924 19:26:52.441340 24547 sys_catalog.cc:564] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:52.441636 24554 sys_catalog.cc:455] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db38f22395da4f9baa8c7c82d607c7f4. Latest consensus state: current_term: 1 leader_uuid: "db38f22395da4f9baa8c7c82d607c7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38f22395da4f9baa8c7c82d607c7f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46257 } } }
I20260924 19:26:52.441742 24554 sys_catalog.cc:458] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:52.441947 24557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:52.441967 24553 sys_catalog.cc:455] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db38f22395da4f9baa8c7c82d607c7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38f22395da4f9baa8c7c82d607c7f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46257 } } }
I20260924 19:26:52.442051 24553 sys_catalog.cc:458] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:52.442606 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.437184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.18.229.190:46257)
0924 19:26:52.437365 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.437373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.437398 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:26:52.437826 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.437833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.437892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.438006 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.438268 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.439088 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.440263 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.440674 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.440765 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.441566 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.441569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.441570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.441792 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.441839 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.441856 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.441928 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.442278 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.442355 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 19:26:52.442435 24557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:52.443429 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:26:52.444941 24557 catalog_manager.cc:1347] Generated new cluster ID: 605356d28eb1443b969cdfa5e40d44a9
I20260924 19:26:52.444993 24557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:52.460438 24557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:52.460968 24557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:52.467964 24557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db38f22395da4f9baa8c7c82d607c7f4: Generated new TSK 0
I20260924 19:26:52.468173 24557 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:26:52.496472 24577 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 19:26:52.496681 24577 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:26:52.496759 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.473689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.18.229.190:46257)
0924 19:26:52.473744 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.473746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.473889 (+ 143us) server_negotiation.cc:407] Connection header received
0924 19:26:52.473990 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.473991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.474001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.474057 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:26:52.474224 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.474693 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.475809 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.475996 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.476045 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.476307 (+ 262us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:26:52.496476 (+ 20169us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:26:52.496664 (+ 188us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53524: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 19:26:52.497062 24551 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53524: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 19:26:52.497084 24576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:52.473613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:46257 (local address 127.0.0.1:53524)
0924 19:26:52.473815 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.473826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.473899 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.474086 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.474087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.474089 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:26:52.474162 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:52.474164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.474762 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.474763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:52.475730 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:52.475733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.476074 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.476075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:52.476178 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.476201 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:26:52.496625 (+ 20424us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:26:52.496752 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.229.190:46257: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.229.190:46257: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 19:26:52.497591 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24480
2026-09-24T19:26:52Z chronyd exiting
I20260924 19:26:52.533599 19350 test_util.cc:175] -----------------------------------------------
I20260924 19:26:52.533677 19350 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790277987605586-19350-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 19:26:52.534418 19350 mini_oidc.cc:152] Starting JWKS server
I20260924 19:26:52.535118 19350 webserver.cc:460] Webserver started at https://127.0.0.1:35475/ using document root <none> and password file <none>
I20260924 19:26:52.535187 19350 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:26:52.535353 19350 webserver.cc:460] Webserver started at http://127.0.0.1:36821/ using document root <none> and password file <none>
2026-09-24T19:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:52Z Disabled control of system clock
I20260924 19:26:52.542635 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:35139
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:35139
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35475/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/testchainca.cert with env {}
W20260924 19:26:52.620805 24583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:52.621035 24583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:52.621069 24583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:52.622211 24583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:52.622280 24583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:52.622300 24583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:52.622321 24583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:52.622351 24583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:26:52.622375 24583 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:26:52.622394 24583 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35475/jwks.json
I20260924 19:26:52.623875 24583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:35139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:35139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35475/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:52.624152 24583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:52.624389 24583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:52.629379 24591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.629442 24583 server_base.cc:1034] running on GCE node
W20260924 19:26:52.629386 24589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:52.629387 24588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.629850 24583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:52.630137 24583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:52.631286 24583 hybrid_clock.cc:648] HybridClock initialized: now 1790278012631273 us; error 30 us; skew 500 ppm
I20260924 19:26:52.632512 24583 webserver.cc:460] Webserver started at http://127.18.229.190:35269/ using document root <none> and password file <none>
I20260924 19:26:52.632723 24583 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:52.632768 24583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:52.632854 24583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:52.633975 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "68a89dde4c55439ca8cc66ec45a0d625"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "3f9d2801de4ac6db795f7117e03c7d8d"
server_key_iv: "755781b55b3507d25c5e97c2f2591ade"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.634289 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "68a89dde4c55439ca8cc66ec45a0d625"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "3f9d2801de4ac6db795f7117e03c7d8d"
server_key_iv: "755781b55b3507d25c5e97c2f2591ade"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.635653 24583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:52.636391 24597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.636569 24583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:52.636644 24583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "68a89dde4c55439ca8cc66ec45a0d625"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "3f9d2801de4ac6db795f7117e03c7d8d"
server_key_iv: "755781b55b3507d25c5e97c2f2591ade"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.636699 24583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:52.649945 24583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:52.650210 24583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:52.650309 24583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:52.654101 24583 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:35139
I20260924 19:26:52.654126 24649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:35139 every 8 connection(s)
I20260924 19:26:52.654461 24583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:52.655087 24650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:52.656543 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24583
I20260924 19:26:52.656620 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:52.656821 19350 external_mini_cluster.cc:1428] Setting key 15b7022bf460ecf153755b3dca1657a7
I20260924 19:26:52.658532 24650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625: Bootstrap starting.
I20260924 19:26:52.659222 24650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:52.659528 24650 log.cc:824] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:52.660260 24650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625: No bootstrap required, opened a new log
I20260924 19:26:52.662336 24650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a89dde4c55439ca8cc66ec45a0d625" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35139 } }
I20260924 19:26:52.662506 24650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:52.662547 24650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a89dde4c55439ca8cc66ec45a0d625, State: Initialized, Role: FOLLOWER
I20260924 19:26:52.662706 24650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a89dde4c55439ca8cc66ec45a0d625" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35139 } }
I20260924 19:26:52.662801 24650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:52.662844 24650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:52.662912 24650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:52.662998 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.657342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.18.229.190:35139)
0924 19:26:52.657545 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.657555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.657587 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:26:52.658247 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.658256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.658346 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.658498 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.658687 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.660048 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.660860 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.661476 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.661594 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.662214 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.662218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.662219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.662427 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.662468 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.662482 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.662552 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.662668 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.662748 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260924 19:26:52.663420 24650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a89dde4c55439ca8cc66ec45a0d625" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35139 } }
I20260924 19:26:52.663532 24650 leader_election.cc:302] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68a89dde4c55439ca8cc66ec45a0d625; no voters:
I20260924 19:26:52.663727 24650 leader_election.cc:288] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:52.663755 24655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:52.663877 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:26:52.663959 24655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [term 1 LEADER]: Becoming Leader. State: Replica: 68a89dde4c55439ca8cc66ec45a0d625, State: Running, Role: LEADER
I20260924 19:26:52.664031 24650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:52.664031 24655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a89dde4c55439ca8cc66ec45a0d625" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35139 } }
I20260924 19:26:52.664431 24656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68a89dde4c55439ca8cc66ec45a0d625. Latest consensus state: current_term: 1 leader_uuid: "68a89dde4c55439ca8cc66ec45a0d625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a89dde4c55439ca8cc66ec45a0d625" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35139 } } }
I20260924 19:26:52.664433 24657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68a89dde4c55439ca8cc66ec45a0d625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a89dde4c55439ca8cc66ec45a0d625" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35139 } } }
I20260924 19:26:52.664531 24656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:52.664558 24657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:52.665354 24662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:52.665820 24662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:52.667250 24662 catalog_manager.cc:1347] Generated new cluster ID: c094eaffc19b4f34a1d5e5c20354e0f2
I20260924 19:26:52.667301 24662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:52.688525 24662 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:52.689292 24662 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:52.695457 24662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68a89dde4c55439ca8cc66ec45a0d625: Generated new TSK 0
I20260924 19:26:52.695648 24662 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:26:52.720428 24680 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 19:26:52.720517 24680 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:26:52.720834 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.706818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.18.229.190:35139)
0924 19:26:52.706883 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.706884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.706892 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:52.706919 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.706919 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.706930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.706987 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:26:52.707133 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.707587 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.708758 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.708939 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.708990 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.709166 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:26:52.720633 (+ 11467us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:26:52.720780 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 19:26:52.720901 24653 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 19:26:52.721019 24679 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:52.706478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:35139 (local address 127.0.0.1:40702)
0924 19:26:52.706638 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.706649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.706705 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.707006 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.707007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.707009 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:26:52.707067 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:52.707068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.707674 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.707675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:52.708671 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:52.708674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.709011 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.709012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:52.709070 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.709076 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:26:52.720774 (+ 11698us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:26:52.720912 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.229.190:35139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260924 19:26:52.721195 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24583
2026-09-24T19:26:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 19:26:52.758793 19350 mini_oidc.cc:152] Starting JWKS server
I20260924 19:26:52.759483 19350 webserver.cc:460] Webserver started at https://127.0.0.1:44777/ using document root <none> and password file <none>
I20260924 19:26:52.759547 19350 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:26:52.759702 19350 webserver.cc:460] Webserver started at http://127.0.0.1:42335/ using document root <none> and password file <none>
2026-09-24T19:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:52Z Disabled control of system clock
I20260924 19:26:52.767132 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:46671
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:46671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44777/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:26:52.845180 24686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:52.845391 24686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:52.845412 24686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:52.846596 24686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:52.846635 24686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:52.846648 24686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:52.846661 24686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:52.846714 24686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:26:52.846730 24686 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:26:52.846753 24686 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44777/jwks.json
I20260924 19:26:52.848083 24686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:46671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44777/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:52.848340 24686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:52.848604 24686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:52.853196 24692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:52.853235 24694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.853329 24686 server_base.cc:1034] running on GCE node
W20260924 19:26:52.853214 24691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:52.853641 24686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:52.853928 24686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:52.855093 24686 hybrid_clock.cc:648] HybridClock initialized: now 1790278012855079 us; error 43 us; skew 500 ppm
I20260924 19:26:52.856329 24686 webserver.cc:460] Webserver started at http://127.18.229.190:36885/ using document root <none> and password file <none>
I20260924 19:26:52.856552 24686 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:52.856618 24686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:52.856775 24686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:52.857847 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "32601ea6af434e50806bbe6099bbe296"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "510c6ab6f3b0aeedea41b4a2d06dadec"
server_key_iv: "bb118d18c9e805b4120384b74544e070"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.858199 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "32601ea6af434e50806bbe6099bbe296"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "510c6ab6f3b0aeedea41b4a2d06dadec"
server_key_iv: "bb118d18c9e805b4120384b74544e070"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.859583 24686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:52.860332 24700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:52.860532 24686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:52.860630 24686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "32601ea6af434e50806bbe6099bbe296"
format_stamp: "Formatted at 2026-09-24 19:26:52 on dist-test-slave-r8cc"
server_key: "510c6ab6f3b0aeedea41b4a2d06dadec"
server_key_iv: "bb118d18c9e805b4120384b74544e070"
server_key_version: "encryptionkey@0"
I20260924 19:26:52.860716 24686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:52.868880 24686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:52.869149 24686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:52.869292 24686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:52.873145 24686 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:46671
I20260924 19:26:52.873175 24752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:46671 every 8 connection(s)
I20260924 19:26:52.873505 24686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:52.874114 24753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:52.876850 24753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296: Bootstrap starting.
I20260924 19:26:52.877406 24753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:52.877684 24753 log.cc:824] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:52.878330 24753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296: No bootstrap required, opened a new log
I20260924 19:26:52.879954 24753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32601ea6af434e50806bbe6099bbe296" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46671 } }
I20260924 19:26:52.880086 24753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:52.880160 24753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32601ea6af434e50806bbe6099bbe296, State: Initialized, Role: FOLLOWER
I20260924 19:26:52.880302 24753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32601ea6af434e50806bbe6099bbe296" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46671 } }
I20260924 19:26:52.880381 24753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:52.880404 24753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:52.880432 24753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:52.880950 24753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32601ea6af434e50806bbe6099bbe296" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46671 } }
I20260924 19:26:52.881031 24753 leader_election.cc:302] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32601ea6af434e50806bbe6099bbe296; no voters:
I20260924 19:26:52.881145 24753 leader_election.cc:288] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:52.881263 24756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:52.881320 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24686
I20260924 19:26:52.881399 24753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:52.881417 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:52.881455 24756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [term 1 LEADER]: Becoming Leader. State: Replica: 32601ea6af434e50806bbe6099bbe296, State: Running, Role: LEADER
I20260924 19:26:52.881560 24756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32601ea6af434e50806bbe6099bbe296" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46671 } }
I20260924 19:26:52.881606 19350 external_mini_cluster.cc:1428] Setting key 7b26409cd99a84c7c06b9e88fa4787c6
I20260924 19:26:52.882277 24758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32601ea6af434e50806bbe6099bbe296. Latest consensus state: current_term: 1 leader_uuid: "32601ea6af434e50806bbe6099bbe296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32601ea6af434e50806bbe6099bbe296" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46671 } } }
I20260924 19:26:52.882360 24758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:52.882685 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32601ea6af434e50806bbe6099bbe296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32601ea6af434e50806bbe6099bbe296" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46671 } } }
I20260924 19:26:52.882766 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296 [sys.catalog]: This master's current role is: LEADER
W20260924 19:26:52.883530 24773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:52.883582 24773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:52.883636 24767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:52.883803 24767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:52.885253 24767 catalog_manager.cc:1347] Generated new cluster ID: e427dcc3f5814ef383f4af715d6fe186
I20260924 19:26:52.885303 24767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:52.887921 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.882049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.18.229.190:46671)
0924 19:26:52.882230 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.882248 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.882440 (+ 192us) server_negotiation.cc:407] Connection header received
0924 19:26:52.882666 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.882676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.882749 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.882907 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:52.883511 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.884789 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.885874 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.886448 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.886556 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.887246 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:52.887250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:52.887250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:52.887445 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:52.887484 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:52.887497 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:52.887555 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:52.887660 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:26:52.887731 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260924 19:26:52.888837 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:26:52.893947 24767 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:52.894431 24767 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:52.906236 24767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32601ea6af434e50806bbe6099bbe296: Generated new TSK 0
I20260924 19:26:52.906440 24767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:26:52.920802 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:52.917563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.18.229.190:46671)
0924 19:26:52.917618 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:52.917620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:52.917726 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:26:52.917806 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:52.917807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:52.917818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:52.917902 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:26:52.918039 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.918533 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.919320 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.919528 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.919614 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.919797 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:26:52.920583 (+ 786us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:26:52.920725 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41150: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 19:26:52.920864 24765 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41150: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 19:26:52.920878 24782 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:52.917491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:46671 (local address 127.0.0.1:41150)
0924 19:26:52.917661 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:52.917670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:52.917744 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:52.917901 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:52.917902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:52.917903 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:26:52.917969 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:52.917971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.918627 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.918628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:52.919226 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:52.919227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:52.919620 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:52.919622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:52.919685 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:52.919690 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:26:52.920689 (+ 999us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:26:52.920777 (+ 88us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.229.190:46671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260924 19:26:52.921064 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24686
E20260924 19:26:52.933527 24783 webserver.cc:533] Webserver: sslize failed:
E20260924 19:26:52.933593 24783 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T19:26:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (200 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T19:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:52Z Disabled control of system clock
I20260924 19:26:52.966563 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42345
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42345
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/testchainca.cert with env {}
W20260924 19:26:53.045168 24787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:53.045414 24787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:53.045452 24787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:53.046612 24787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:53.046684 24787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:53.046707 24787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:53.046736 24787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:53.046783 24787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:26:53.046808 24787 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:26:53.046832 24787 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 19:26:53.048360 24787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:53.048637 24787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:53.048857 24787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:53.053377 24792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:53.053407 24795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:53.053478 24793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:53.053637 24787 server_base.cc:1034] running on GCE node
I20260924 19:26:53.053953 24787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:53.054273 24787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:53.055435 24787 hybrid_clock.cc:648] HybridClock initialized: now 1790278013055430 us; error 36 us; skew 500 ppm
I20260924 19:26:53.056622 24787 webserver.cc:460] Webserver started at http://127.18.229.190:44779/ using document root <none> and password file <none>
I20260924 19:26:53.056869 24787 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:53.056937 24787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:53.057083 24787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:53.058099 24787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "3a300f91be6642478aee8ac72dbc315d"
format_stamp: "Formatted at 2026-09-24 19:26:53 on dist-test-slave-r8cc"
server_key: "a1adfb94b1b064c297b2cd897a523673"
server_key_iv: "b24e06466047082ef125f20e73896581"
server_key_version: "encryptionkey@0"
I20260924 19:26:53.058460 24787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "3a300f91be6642478aee8ac72dbc315d"
format_stamp: "Formatted at 2026-09-24 19:26:53 on dist-test-slave-r8cc"
server_key: "a1adfb94b1b064c297b2cd897a523673"
server_key_iv: "b24e06466047082ef125f20e73896581"
server_key_version: "encryptionkey@0"
I20260924 19:26:53.059755 24787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:26:53.060508 24801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:53.060671 24787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:53.060768 24787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "3a300f91be6642478aee8ac72dbc315d"
format_stamp: "Formatted at 2026-09-24 19:26:53 on dist-test-slave-r8cc"
server_key: "a1adfb94b1b064c297b2cd897a523673"
server_key_iv: "b24e06466047082ef125f20e73896581"
server_key_version: "encryptionkey@0"
I20260924 19:26:53.060848 24787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:53.081923 24787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:53.082286 24787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:53.082453 24787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:53.086347 24787 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42345
I20260924 19:26:53.086402 24853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42345 every 8 connection(s)
I20260924 19:26:53.086748 24787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:53.087751 24854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:53.090035 24854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d: Bootstrap starting.
I20260924 19:26:53.090644 24854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:53.090695 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24787
I20260924 19:26:53.090807 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:53.091001 19350 external_mini_cluster.cc:1428] Setting key 8b87d1be9b9a4ee8bd98e7a350781c59
I20260924 19:26:53.091001 24854 log.cc:824] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:53.091830 24854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d: No bootstrap required, opened a new log
I20260924 19:26:53.094334 24854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a300f91be6642478aee8ac72dbc315d" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42345 } }
I20260924 19:26:53.094496 24854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:53.094533 24854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a300f91be6642478aee8ac72dbc315d, State: Initialized, Role: FOLLOWER
I20260924 19:26:53.094655 24854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a300f91be6642478aee8ac72dbc315d" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42345 } }
I20260924 19:26:53.094743 24854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:53.094779 24854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:53.094825 24854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:53.095492 24854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a300f91be6642478aee8ac72dbc315d" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42345 } }
I20260924 19:26:53.095593 24854 leader_election.cc:302] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a300f91be6642478aee8ac72dbc315d; no voters:
I20260924 19:26:53.095762 24854 leader_election.cc:288] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:53.095788 24859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:53.095885 24859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [term 1 LEADER]: Becoming Leader. State: Replica: 3a300f91be6642478aee8ac72dbc315d, State: Running, Role: LEADER
I20260924 19:26:53.095965 24859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a300f91be6642478aee8ac72dbc315d" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42345 } }
I20260924 19:26:53.096011 24854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:53.096297 24859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a300f91be6642478aee8ac72dbc315d. Latest consensus state: current_term: 1 leader_uuid: "3a300f91be6642478aee8ac72dbc315d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a300f91be6642478aee8ac72dbc315d" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42345 } } }
I20260924 19:26:53.096364 24859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:53.096581 24864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:53.096611 24860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a300f91be6642478aee8ac72dbc315d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a300f91be6642478aee8ac72dbc315d" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42345 } } }
I20260924 19:26:53.096781 24860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:53.097100 24864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:53.097970 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:53.091504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.18.229.190:42345)
0924 19:26:53.091701 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:53.091710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:53.092319 (+ 609us) server_negotiation.cc:407] Connection header received
0924 19:26:53.093131 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:53.093139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:53.093200 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:53.093318 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:53.093567 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:53.094417 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:53.095479 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:53.096086 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:53.096258 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:53.097051 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:53.097055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:53.097056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:26:53.097310 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:53.097430 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:26:53.097448 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:53.097688 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:53.097720 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 19:26:53.097797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 19:26:53.098992 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:26:53.100517 24864 catalog_manager.cc:1347] Generated new cluster ID: 4e69266be1be4d5babeaf82cbdaf2b58
I20260924 19:26:53.100572 24864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:53.118692 24864 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:53.119220 24864 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:53.126847 24864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a300f91be6642478aee8ac72dbc315d: Generated new TSK 0
I20260924 19:26:53.127056 24864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:26:53.632905 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:53.132914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.18.229.190:42345)
0924 19:26:53.132958 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:53.132960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:53.133044 (+ 84us) server_negotiation.cc:407] Connection header received
0924 19:26:53.133137 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:53.133137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:53.133149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:53.133207 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:26:53.133345 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:53.133818 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:53.134923 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:53.135107 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:53.135159 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:53.135337 (+ 178us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:26:53.632688 (+497351us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:26:53.632842 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56556: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:26:53.632998 24858 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56556: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 19:26:53.633060 24883 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:53.132819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42345 (local address 127.0.0.1:56556)
0924 19:26:53.132983 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:53.132995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:53.133064 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:53.133222 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:53.133223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:53.133226 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:26:53.133284 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:53.133286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:53.133892 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:53.133892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:53.134844 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:53.134847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:53.135184 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:53.135185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:53.135244 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:53.135249 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:26:53.632838 (+497589us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:26:53.632952 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.229.190:42345: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":151,"threads_started":1}
I20260924 19:26:53.633368 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24787
2026-09-24T19:26:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (691 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:54 dist-test-slave-r8cc krb5kdc[24895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:54 dist-test-slave-r8cc krb5kdc[24895](info): set up 2 sockets
Sep 24 19:26:54 dist-test-slave-r8cc krb5kdc[24895](info): commencing operation
krb5kdc: starting...
W20260924 19:26:55.548365 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.725s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:55 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:26:55.663404 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42361
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:55.743570 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:55.743788 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:55.743825 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:55.744997 24908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:55.745060 24908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:55.745083 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:55.745103 24908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:55.745133 24908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:55.746567 24908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:55.746838 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:55.747058 24908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:55.751675 24913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:55.751657 24916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:55.751762 24908 server_base.cc:1034] running on GCE node
W20260924 19:26:55.751647 24914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:55.752233 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:55.752523 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:55.753710 24908 hybrid_clock.cc:648] HybridClock initialized: now 1790278015753696 us; error 27 us; skew 500 ppm
Sep 24 19:26:55 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278015, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:55.756232 24908 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:26:55.756651 24908 webserver.cc:460] Webserver started at http://127.18.229.190:39667/ using document root <none> and password file <none>
I20260924 19:26:55.756914 24908 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:55.756981 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:55.757129 24908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:55.758160 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "f265bbe35d72441e8d23a8ec2e810251"
format_stamp: "Formatted at 2026-09-24 19:26:55 on dist-test-slave-r8cc"
server_key: "c8fec68f8ee7ea2023c5e0f75e9e9d18"
server_key_iv: "002aedf493df9e9af6b93487cb590e1a"
server_key_version: "encryptionkey@0"
I20260924 19:26:55.758513 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "f265bbe35d72441e8d23a8ec2e810251"
format_stamp: "Formatted at 2026-09-24 19:26:55 on dist-test-slave-r8cc"
server_key: "c8fec68f8ee7ea2023c5e0f75e9e9d18"
server_key_iv: "002aedf493df9e9af6b93487cb590e1a"
server_key_version: "encryptionkey@0"
I20260924 19:26:55.759788 24908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:55.760543 24923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:55.760716 24908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:55.760808 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "f265bbe35d72441e8d23a8ec2e810251"
format_stamp: "Formatted at 2026-09-24 19:26:55 on dist-test-slave-r8cc"
server_key: "c8fec68f8ee7ea2023c5e0f75e9e9d18"
server_key_iv: "002aedf493df9e9af6b93487cb590e1a"
server_key_version: "encryptionkey@0"
I20260924 19:26:55.760887 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:55.770514 24908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:55.772779 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:55.772938 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:55.776603 24908 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42361
I20260924 19:26:55.776638 24975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42361 every 8 connection(s)
I20260924 19:26:55.776975 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:55.777843 24976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:55.777911 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 24908
I20260924 19:26:55.778041 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:55.778256 19350 external_mini_cluster.cc:1428] Setting key e2d4eca5a4cdc00a09efcadd74b4b732
I20260924 19:26:55.781164 24976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251: Bootstrap starting.
I20260924 19:26:55.781879 24976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:55.782168 24976 log.cc:824] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:55.782882 24976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251: No bootstrap required, opened a new log
Sep 24 19:26:55 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:55.785146 24976 raft_consensus.cc:348] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f265bbe35d72441e8d23a8ec2e810251" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42361 } }
I20260924 19:26:55.785323 24976 raft_consensus.cc:374] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:55.785360 24976 raft_consensus.cc:729] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f265bbe35d72441e8d23a8ec2e810251, State: Initialized, Role: FOLLOWER
I20260924 19:26:55.785494 24976 consensus_queue.cc:260] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f265bbe35d72441e8d23a8ec2e810251" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42361 } }
I20260924 19:26:55.785578 24976 raft_consensus.cc:388] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:55.785612 24976 raft_consensus.cc:482] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:55.785662 24976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:55.786358 24976 raft_consensus.cc:504] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f265bbe35d72441e8d23a8ec2e810251" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42361 } }
I20260924 19:26:55.786443 24976 leader_election.cc:302] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f265bbe35d72441e8d23a8ec2e810251; no voters:
I20260924 19:26:55.786625 24976 leader_election.cc:288] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:55.786682 24981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:55.786850 24981 raft_consensus.cc:686] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [term 1 LEADER]: Becoming Leader. State: Replica: f265bbe35d72441e8d23a8ec2e810251, State: Running, Role: LEADER
I20260924 19:26:55.786893 24976 sys_catalog.cc:564] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:55.787045 24981 consensus_queue.cc:237] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f265bbe35d72441e8d23a8ec2e810251" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42361 } }
I20260924 19:26:55.788239 24983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f265bbe35d72441e8d23a8ec2e810251. Latest consensus state: current_term: 1 leader_uuid: "f265bbe35d72441e8d23a8ec2e810251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f265bbe35d72441e8d23a8ec2e810251" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42361 } } }
I20260924 19:26:55.788323 24982 sys_catalog.cc:455] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f265bbe35d72441e8d23a8ec2e810251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f265bbe35d72441e8d23a8ec2e810251" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42361 } } }
I20260924 19:26:55.788335 24983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [sys.catalog]: This master's current role is: LEADER
W20260924 19:26:55.788900 24996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:55.789005 24996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:55.788905 24982 sys_catalog.cc:458] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251 [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:55.789455 24998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:55.789616 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:55.778866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.18.229.190:42361)
0924 19:26:55.779021 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:55.779028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:55.779050 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:26:55.779475 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:55.779482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:55.779526 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:55.779637 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:55.780473 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:55.781398 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:55.782465 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:55.782977 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:55.783122 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:55.785740 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:55.785744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:55.785744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:55.785757 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:55.787749 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:55.788187 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:55.788190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:55.788190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:55.788302 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:55.788487 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:55.788488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:55.788488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:55.788800 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:55.788964 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:55.789247 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:26:55.789386 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 19:26:55.789664 24998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:55.791011 24998 catalog_manager.cc:1347] Generated new cluster ID: c3fea36b4eee419f91d1cf96a9a7bc76
I20260924 19:26:55.791064 24998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:26:55.808626 24998 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:55.809144 24998 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:55.812130 24998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f265bbe35d72441e8d23a8ec2e810251: Generated new TSK 0
I20260924 19:26:55.812316 24998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:26:55.840742 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42361
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:55.923609 25002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:55.923799 25002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:55.923821 25002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:55.925083 25002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:55.925125 25002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:55.925184 25002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:55.926640 25002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:55.926882 25002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:55.927109 25002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:55.931914 25008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:55.931949 25010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:55.931943 25007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:55.932140 25002 server_base.cc:1034] running on GCE node
I20260924 19:26:55.932320 25002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:55.932538 25002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:55.933693 25002 hybrid_clock.cc:648] HybridClock initialized: now 1790278015933678 us; error 33 us; skew 500 ppm
Sep 24 19:26:55 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278015, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:55.936092 25002 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:26:55.936633 25002 webserver.cc:460] Webserver started at http://127.18.229.129:37293/ using document root <none> and password file <none>
I20260924 19:26:55.936865 25002 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:55.936929 25002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:55.937063 25002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:55.938077 25002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "a06369ee060a425ea72e37511f15ca2a"
format_stamp: "Formatted at 2026-09-24 19:26:55 on dist-test-slave-r8cc"
server_key: "5d75e37cea0ad9981b0e7e068488cd13"
server_key_iv: "0b3614790337a5b5d3a85593831cf62b"
server_key_version: "encryptionkey@0"
I20260924 19:26:55.938426 25002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "a06369ee060a425ea72e37511f15ca2a"
format_stamp: "Formatted at 2026-09-24 19:26:55 on dist-test-slave-r8cc"
server_key: "5d75e37cea0ad9981b0e7e068488cd13"
server_key_iv: "0b3614790337a5b5d3a85593831cf62b"
server_key_version: "encryptionkey@0"
I20260924 19:26:55.939687 25002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:55.940390 25017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:55.940582 25002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:55.940685 25002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "a06369ee060a425ea72e37511f15ca2a"
format_stamp: "Formatted at 2026-09-24 19:26:55 on dist-test-slave-r8cc"
server_key: "5d75e37cea0ad9981b0e7e068488cd13"
server_key_iv: "0b3614790337a5b5d3a85593831cf62b"
server_key_version: "encryptionkey@0"
I20260924 19:26:55.940768 25002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:55.961220 25002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:55.963106 25002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:55.963264 25002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:55.963516 25002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:55.963824 25002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:55.963878 25002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:55.963913 25002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:55.963940 25002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:55.969770 25002 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:38397
I20260924 19:26:55.969861 25130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:38397 every 8 connection(s)
I20260924 19:26:55.970180 25002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:26:55 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278015, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:55.976272 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25002
I20260924 19:26:55.976388 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:55.976604 19350 external_mini_cluster.cc:1428] Setting key 775fc956c020f3b23124542caea2e739
I20260924 19:26:55.979251 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:55.970892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:47979 (local address 127.18.229.190:42361)
0924 19:26:55.970952 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:55.970954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:55.971171 (+ 217us) server_negotiation.cc:407] Connection header received
0924 19:26:55.972069 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:55.972070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:55.972080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:55.972190 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:55.972801 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:55.973295 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:55.974223 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:55.974401 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:55.974448 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:55.976506 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:55.976507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:55.976508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:55.976515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:55.977898 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:55.978305 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:55.978306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:55.978307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:55.978369 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:55.978614 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:55.978615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:55.978616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:55.978736 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:55.978812 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:55.979076 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:26:55.979153 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:26:55.979462 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:55.970859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42361 (local address 127.18.229.129:47979)
0924 19:26:55.971040 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:55.971079 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:55.971983 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:55.972262 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:55.972265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:55.972500 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:55.972725 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:55.972729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:55.973368 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:55.973369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:55.974153 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:55.974155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:55.974467 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:55.974468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:55.974584 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:55.975026 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:55.975034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:55.976393 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:55.977978 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:55.977985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:55.977987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:55.978216 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:55.978435 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:55.978436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:55.978437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:55.978547 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:55.978804 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:55.978811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:55.978945 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:26:55.979055 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:26:55.979201 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":114,"threads_started":1}
I20260924 19:26:55.979995 25131 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42361
I20260924 19:26:55.980095 25131 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:55.980281 25131 heartbeater.cc:511] Master 127.18.229.190:42361 requested a full tablet report, sending...
I20260924 19:26:55.980870 24940 ts_manager.cc:194] Registered new tserver with Master: a06369ee060a425ea72e37511f15ca2a (127.18.229.129:38397)
I20260924 19:26:55.981827 24940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:47979
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:26:56.000846 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42361
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:56.081065 25136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:56.081274 25136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:56.081297 25136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:56.082470 25136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:56.082506 25136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:56.082533 25136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:56.083942 25136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:56.084194 25136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:56.084378 25136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:56.089685 25142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:56.089692 25141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:56.089709 25136 server_base.cc:1034] running on GCE node
W20260924 19:26:56.089692 25144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:56.090150 25136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:56.090415 25136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:56.091576 25136 hybrid_clock.cc:648] HybridClock initialized: now 1790278016091559 us; error 29 us; skew 500 ppm
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.094025 25136 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:26:56.094440 25136 webserver.cc:460] Webserver started at http://127.18.229.130:40543/ using document root <none> and password file <none>
I20260924 19:26:56.094651 25136 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:56.094722 25136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:56.094866 25136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:56.095837 25136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "ae5746a400444bf1a59ed90fe9def550"
format_stamp: "Formatted at 2026-09-24 19:26:56 on dist-test-slave-r8cc"
server_key: "b3074310d96b5add63ff24c89b05a7fc"
server_key_iv: "8311cd90ba4300c7d3f63504c7da6700"
server_key_version: "encryptionkey@0"
I20260924 19:26:56.096236 25136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5746a400444bf1a59ed90fe9def550"
format_stamp: "Formatted at 2026-09-24 19:26:56 on dist-test-slave-r8cc"
server_key: "b3074310d96b5add63ff24c89b05a7fc"
server_key_iv: "8311cd90ba4300c7d3f63504c7da6700"
server_key_version: "encryptionkey@0"
I20260924 19:26:56.097523 25136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:56.098255 25151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:56.098424 25136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:56.098517 25136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "ae5746a400444bf1a59ed90fe9def550"
format_stamp: "Formatted at 2026-09-24 19:26:56 on dist-test-slave-r8cc"
server_key: "b3074310d96b5add63ff24c89b05a7fc"
server_key_iv: "8311cd90ba4300c7d3f63504c7da6700"
server_key_version: "encryptionkey@0"
I20260924 19:26:56.098598 25136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:56.109104 25136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:56.110771 25136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:56.110919 25136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:56.111132 25136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:56.111413 25136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:56.111464 25136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:56.111503 25136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:56.111529 25136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:56.117532 25136 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:40199
I20260924 19:26:56.117578 25264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:40199 every 8 connection(s)
I20260924 19:26:56.117870 25136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:56.125533 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25136
I20260924 19:26:56.125655 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:56.125850 19350 external_mini_cluster.cc:1428] Setting key 992d693af34170f749d50ee2b12f8dd6
I20260924 19:26:56.128566 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.118517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:46419 (local address 127.18.229.190:42361)
0924 19:26:56.118559 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.118561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.118748 (+ 187us) server_negotiation.cc:407] Connection header received
0924 19:26:56.119838 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.119839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.119854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.119924 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.120720 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.121475 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.122677 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.122967 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.123077 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.125579 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:56.125581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:56.125581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:56.125589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:56.127150 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.127575 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.127577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.127577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.127639 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.127897 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.127898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.127898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.128011 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:56.128083 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:56.128391 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:26:56.128480 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:26:56.128746 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.118486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42361 (local address 127.18.229.130:46419)
0924 19:26:56.118623 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:56.118661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:56.119685 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:56.119999 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:56.120003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:56.120404 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:56.120632 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:56.120637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.121624 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.121625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:56.122521 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:56.122523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.123151 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.123152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:56.123252 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.123772 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:56.123789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:56.125480 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:56.127251 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:56.127257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:56.127259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:56.127488 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:56.127705 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:56.127706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:56.127707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:56.127819 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:56.128088 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:56.128094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:56.128248 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 19:26:56.128360 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:26:56.128509 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260924 19:26:56.129279 25265 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42361
I20260924 19:26:56.129357 25265 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:56.129555 25265 heartbeater.cc:511] Master 127.18.229.190:42361 requested a full tablet report, sending...
I20260924 19:26:56.129909 24940 ts_manager.cc:194] Registered new tserver with Master: ae5746a400444bf1a59ed90fe9def550 (127.18.229.130:40199)
I20260924 19:26:56.130332 24940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:46419
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:26:56.148653 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42361
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:56.227490 25270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:56.227717 25270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:56.227757 25270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:56.228910 25270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:56.228976 25270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:56.229017 25270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:56.230446 25270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:56.230720 25270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:56.230933 25270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:26:56.235656 25270 server_base.cc:1034] running on GCE node
W20260924 19:26:56.235613 25278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:56.235613 25276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:56.235630 25275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:56.236088 25270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:56.236336 25270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:56.237460 25270 hybrid_clock.cc:648] HybridClock initialized: now 1790278016237452 us; error 30 us; skew 500 ppm
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.239676 25270 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:26:56.240025 25270 webserver.cc:460] Webserver started at http://127.18.229.131:44653/ using document root <none> and password file <none>
I20260924 19:26:56.240278 25270 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:56.240324 25270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:56.240408 25270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:56.241415 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "aad4c71ab75b43d790ca017e4d57f4f7"
format_stamp: "Formatted at 2026-09-24 19:26:56 on dist-test-slave-r8cc"
server_key: "21f61014ba57c8916e5eee0e3caa6fa0"
server_key_iv: "e10d7b987d5c5f8ba12f31eff2ad2408"
server_key_version: "encryptionkey@0"
I20260924 19:26:56.241709 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "aad4c71ab75b43d790ca017e4d57f4f7"
format_stamp: "Formatted at 2026-09-24 19:26:56 on dist-test-slave-r8cc"
server_key: "21f61014ba57c8916e5eee0e3caa6fa0"
server_key_iv: "e10d7b987d5c5f8ba12f31eff2ad2408"
server_key_version: "encryptionkey@0"
I20260924 19:26:56.242892 25270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:56.243554 25285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:56.243701 25270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:56.243803 25270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "aad4c71ab75b43d790ca017e4d57f4f7"
format_stamp: "Formatted at 2026-09-24 19:26:56 on dist-test-slave-r8cc"
server_key: "21f61014ba57c8916e5eee0e3caa6fa0"
server_key_iv: "e10d7b987d5c5f8ba12f31eff2ad2408"
server_key_version: "encryptionkey@0"
I20260924 19:26:56.243876 25270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:56.254277 25270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:56.255865 25270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:56.255966 25270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:56.256182 25270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:56.256438 25270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:56.256466 25270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:56.256485 25270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:56.256497 25270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:56.262072 25270 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:36905
I20260924 19:26:56.262144 25398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:36905 every 8 connection(s)
I20260924 19:26:56.262449 25270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:56.263092 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25270
I20260924 19:26:56.263192 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:56.263427 19350 external_mini_cluster.cc:1428] Setting key 0bdc3a3e907de2bb4474c4241680458a
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:26:56.270982 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.263344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:56439 (local address 127.18.229.190:42361)
0924 19:26:56.263414 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.263416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.263428 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:26:56.264630 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.264630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.264640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.264682 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.265222 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.265698 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.266600 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.266774 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.266821 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.268793 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:56.268795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:56.268795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:56.268801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:56.269931 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.270256 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.270256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.270257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.270304 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.270498 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.270499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.270499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.270586 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:56.270636 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:56.270867 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:26:56.270922 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:26:56.271127 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.263112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42361 (local address 127.18.229.131:56439)
0924 19:26:56.263257 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:56.263296 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:56.264545 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:56.264745 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:56.264748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:56.264981 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:56.265149 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:56.265152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.265773 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.265774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:56.266531 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:56.266533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.266837 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.266838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:56.266926 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.267356 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:56.267364 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:56.268704 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:56.269995 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:56.270001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:56.270002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:56.270195 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:56.270357 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:56.270357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:56.270357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:56.270452 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:56.270638 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:56.270641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:56.270764 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 19:26:56.270841 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:26:56.270945 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260924 19:26:56.271569 25399 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42361
I20260924 19:26:56.271643 25399 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:56.271834 25399 heartbeater.cc:511] Master 127.18.229.190:42361 requested a full tablet report, sending...
I20260924 19:26:56.272264 24940 ts_manager.cc:194] Registered new tserver with Master: aad4c71ab75b43d790ca017e4d57f4f7 (127.18.229.131:36905)
I20260924 19:26:56.272635 24940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:56439
I20260924 19:26:56.275195 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/test-admin.keytab.
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.288394 19350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 19:26:56.290045 25406 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:56.289262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.18.229.129:38397)
0924 19:26:56.289417 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.289422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.289435 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:26:56.289467 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.289470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.289487 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.289585 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.289946 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33982: BlockingRecv error: recv got EOF from 127.0.0.1:33982 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260924 19:26:56.290066 19350 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.291013 19350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.291846 19350 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:56.298913 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.292421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.18.229.129:38397)
0924 19:26:56.292463 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.292465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.292492 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:26:56.292544 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.292545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.292553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.292609 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.292953 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.293652 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.294313 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.294602 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.294653 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.296636 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:56.296642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:56.296642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:56.296648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:56.298011 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.298312 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.298314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.298315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.298364 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.298518 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.298518 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.298518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.298630 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:56.298724 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:56.298789 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:26:56.298845 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:26:56.300671 25406 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:56.300183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.18.229.129:38397)
0924 19:26:56.300234 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.300236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.300277 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:26:56.300324 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.300324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.300332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.300378 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.300601 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34008: BlockingRecv error: recv got EOF from 127.0.0.1:34008 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 19:26:56.300786 19350 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.301720 19350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.302640 19350 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:56.308776 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.303080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.18.229.129:38397)
0924 19:26:56.303112 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.303114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.303156 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:26:56.303201 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.303202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.303210 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.303258 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.303564 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.304020 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.304697 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.304892 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.304940 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.306771 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:56.306772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:56.306772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:56.306777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:56.307902 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.308248 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.308249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.308249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.308293 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.308439 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.308439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.308440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.308509 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:56.308570 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:56.308644 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 19:26:56.308724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:26:56.310177 25406 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:26:56.309710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.18.229.129:38397)
0924 19:26:56.309747 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.309749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.309792 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:26:56.309846 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.309846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.309853 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.309902 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.310112 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34016: BlockingRecv error: recv got EOF from 127.0.0.1:34016 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 19:26:56.310290 19350 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790277987605586-19350-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.311153 19350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:56.312013 19350 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:26:56.318106 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:56.312609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.18.229.129:38397)
0924 19:26:56.312643 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:56.312645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:56.312714 (+ 69us) server_negotiation.cc:407] Connection header received
0924 19:26:56.312766 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:56.312767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:56.312774 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:56.312811 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:56.313124 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.313539 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.314140 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:56.314308 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:56.314354 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:56.316256 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:56.316257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:56.316257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:56.316262 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:56.317336 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.317611 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.317612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.317612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.317656 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:56.317797 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:56.317798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:56.317798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:56.317867 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:56.317927 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:56.318009 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:26:56.318056 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:26:56.318742 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25002
I20260924 19:26:56.323652 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25136
I20260924 19:26:56.327754 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25270
I20260924 19:26:56.332052 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 24908
2026-09-24T19:26:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2527 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[25419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[25419](info): set up 2 sockets
Sep 24 19:26:56 dist-test-slave-r8cc krb5kdc[25419](info): commencing operation
krb5kdc: starting...
W20260924 19:26:57.356326 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:26:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab.
Entry for principal oryx/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab.
I20260924 19:26:57.471108 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43627
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab
--principal=oryx/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:57.549551 25432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:57.549777 25432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:57.549813 25432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:57.551103 25432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:57.551158 25432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:57.551182 25432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:57.551226 25432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:57.551254 25432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:57.552812 25432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab
--principal=oryx/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:57.553078 25432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:57.553295 25432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:57.557906 25440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:57.557982 25432 server_base.cc:1034] running on GCE node
W20260924 19:26:57.557960 25437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:57.557945 25438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:57.558377 25432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:57.558630 25432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:57.559773 25432 hybrid_clock.cc:648] HybridClock initialized: now 1790278017559757 us; error 39 us; skew 500 ppm
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:57.562111 25432 init.cc:377] Logged in from keytab as oryx/127.18.229.190@KRBTEST.COM (short username oryx)
I20260924 19:26:57.562605 25432 webserver.cc:460] Webserver started at http://127.18.229.190:42615/ using document root <none> and password file <none>
I20260924 19:26:57.562836 25432 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:57.562899 25432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:57.563040 25432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:57.564077 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "261791e4f0ef4077b728544ae9a8b4af"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "8cf72aa222aceca2c48d35feffd8a460"
server_key_iv: "f19624e891251578ef5878aefe1bdfeb"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.564486 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "261791e4f0ef4077b728544ae9a8b4af"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "8cf72aa222aceca2c48d35feffd8a460"
server_key_iv: "f19624e891251578ef5878aefe1bdfeb"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.565773 25432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:57.566468 25447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.566632 25432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:57.566726 25432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "261791e4f0ef4077b728544ae9a8b4af"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "8cf72aa222aceca2c48d35feffd8a460"
server_key_iv: "f19624e891251578ef5878aefe1bdfeb"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.566803 25432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:57.580286 25432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:57.582103 25432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:57.582259 25432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:57.586292 25432 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:43627
I20260924 19:26:57.586333 25499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:43627 every 8 connection(s)
I20260924 19:26:57.586705 25432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:26:57.587657 25500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:57.589936 25500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Bootstrap starting.
I20260924 19:26:57.590507 25500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:57.590811 25500 log.cc:824] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:57.591482 25500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: No bootstrap required, opened a new log
I20260924 19:26:57.593077 25500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } }
I20260924 19:26:57.593221 25500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:57.593268 25500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261791e4f0ef4077b728544ae9a8b4af, State: Initialized, Role: FOLLOWER
I20260924 19:26:57.593423 25500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } }
I20260924 19:26:57.593485 25500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:26:57.593528 25500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:26:57.593575 25500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:57.594334 25500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } }
I20260924 19:26:57.594446 25500 leader_election.cc:302] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261791e4f0ef4077b728544ae9a8b4af; no voters:
I20260924 19:26:57.594620 25500 leader_election.cc:288] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:26:57.594727 25503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:57.594942 25500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:57.594971 25503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 LEADER]: Becoming Leader. State: Replica: 261791e4f0ef4077b728544ae9a8b4af, State: Running, Role: LEADER
I20260924 19:26:57.595072 25503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } }
I20260924 19:26:57.595511 25504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } }
I20260924 19:26:57.595525 25505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261791e4f0ef4077b728544ae9a8b4af. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } }
I20260924 19:26:57.595709 25505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:57.595629 25504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:57.596146 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25432
I20260924 19:26:57.596284 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:26:57.596491 19350 external_mini_cluster.cc:1428] Setting key a6dd00880886c688eea71fd4d5f28e4a
W20260924 19:26:57.597313 25519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:57.597357 25519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:57.597425 25512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:26:57.597601 25512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:26:57.599202 25512 catalog_manager.cc:1347] Generated new cluster ID: 9e3e6509a910435389bb4c6ff093c579
I20260924 19:26:57.599308 25512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:26:57.606271 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:57.597067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.18.229.190:43627)
0924 19:26:57.597347 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:57.597384 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:57.597420 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:26:57.597550 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:57.597557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:57.597621 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:57.597788 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:57.598509 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.599384 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.600236 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.600651 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.600774 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:57.603045 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:57.603055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:57.603055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:57.603066 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:57.604803 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:57.605175 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:57.605178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:57.605179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:57.605277 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:57.605489 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:57.605490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:57.605491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:57.605726 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:57.605904 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:57.605972 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:26:57.606109 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":77,"threads_started":1}
I20260924 19:26:57.618917 25512 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:26:57.619701 25512 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:26:57.624398 25512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Generated new TSK 0
I20260924 19:26:57.624615 25512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab.
Entry for principal oryx/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab.
I20260924 19:26:57.656447 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43627
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:57.735651 25526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:57.735859 25526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:57.735883 25526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:57.737066 25526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:57.737107 25526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:57.737133 25526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:57.738600 25526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:57.738828 25526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:57.739076 25526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:57.744076 25534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:57.744079 25532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:57.744303 25526 server_base.cc:1034] running on GCE node
W20260924 19:26:57.744064 25531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:57.744556 25526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:57.744814 25526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:57.745955 25526 hybrid_clock.cc:648] HybridClock initialized: now 1790278017745942 us; error 29 us; skew 500 ppm
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:57.748332 25526 init.cc:377] Logged in from keytab as oryx/127.18.229.129@KRBTEST.COM (short username oryx)
I20260924 19:26:57.748746 25526 webserver.cc:460] Webserver started at http://127.18.229.129:33841/ using document root <none> and password file <none>
I20260924 19:26:57.748977 25526 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:57.749049 25526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:57.749212 25526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:57.750239 25526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "f885e65c6bdc477a8baa2b67ba10ce08"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "627890260d093b9009721f01a572d062"
server_key_iv: "6d9ba8fb22c979156f53e857053186f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.750593 25526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "f885e65c6bdc477a8baa2b67ba10ce08"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "627890260d093b9009721f01a572d062"
server_key_iv: "6d9ba8fb22c979156f53e857053186f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.751865 25526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:26:57.752650 25541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.752830 25526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.752925 25526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "f885e65c6bdc477a8baa2b67ba10ce08"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "627890260d093b9009721f01a572d062"
server_key_iv: "6d9ba8fb22c979156f53e857053186f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.753010 25526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:57.770853 25526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:57.772818 25526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:57.772984 25526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:57.773214 25526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:57.773528 25526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:57.773582 25526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.773623 25526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:57.773651 25526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.779150 25526 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:37889
I20260924 19:26:57.779256 25654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:37889 every 8 connection(s)
I20260924 19:26:57.779544 25526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:26:57.780834 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25526
I20260924 19:26:57.780936 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:26:57.781139 19350 external_mini_cluster.cc:1428] Setting key 4852ba0c272311ba2358352b8f58fa48
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.229.130@KRBTEST.COM; defaulting to no policy
I20260924 19:26:57.789968 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:57.780352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:42639 (local address 127.18.229.190:43627)
0924 19:26:57.780423 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:57.780425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:57.780608 (+ 183us) server_negotiation.cc:407] Connection header received
0924 19:26:57.781336 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:57.781337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:57.781352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:57.781418 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:57.782727 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.783211 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.784276 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.784484 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.784562 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:57.787123 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:57.787124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:57.787124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:57.787130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:57.788525 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:57.789004 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:57.789005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:57.789005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:57.789054 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:57.789329 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:57.789330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:57.789330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:57.789423 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:57.789471 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:57.789844 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 19:26:57.789904 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:26:57.790220 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:57.780346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:42639)
0924 19:26:57.780491 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:57.780526 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:57.781219 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:57.781567 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:57.781572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:57.782357 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:57.782633 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:57.782639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.783321 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.783322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:57.784168 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:57.784172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.784571 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.784572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:57.784704 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:57.785273 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:57.785284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:57.787034 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:57.788604 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:57.788611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:57.788614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:57.788929 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:57.789120 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:57.789121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:57.789122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:57.789264 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:57.789484 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:57.789490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:57.789679 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 19:26:57.789821 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 19:26:57.789987 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260924 19:26:57.790761 25655 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43627
I20260924 19:26:57.790859 25655 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:57.791028 25655 heartbeater.cc:511] Master 127.18.229.190:43627 requested a full tablet report, sending...
I20260924 19:26:57.791581 25464 ts_manager.cc:194] Registered new tserver with Master: f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.130@KRBTEST.COM" created.
I20260924 19:26:57.792505 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:42639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab.
Entry for principal oryx/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab.
I20260924 19:26:57.804872 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43627
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:57.886386 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:57.886579 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:57.886601 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:57.887737 25660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:57.887779 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:57.887831 25660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:57.889389 25660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:57.889647 25660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:57.889876 25660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:57.894670 25666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:57.894693 25668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:57.894737 25665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:57.894963 25660 server_base.cc:1034] running on GCE node
I20260924 19:26:57.895179 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:57.895417 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:57.896586 25660 hybrid_clock.cc:648] HybridClock initialized: now 1790278017896573 us; error 25 us; skew 500 ppm
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:57.898898 25660 init.cc:377] Logged in from keytab as oryx/127.18.229.130@KRBTEST.COM (short username oryx)
I20260924 19:26:57.899304 25660 webserver.cc:460] Webserver started at http://127.18.229.130:37045/ using document root <none> and password file <none>
I20260924 19:26:57.899521 25660 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:57.899585 25660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:57.899734 25660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:57.900807 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "52fb1d48dca048658a72aa55b3139c85"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "3582fd9d37e39dca9e821f297d74b18b"
server_key_iv: "7cfe8940c7fa518e608d896a723ceb76"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.901156 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "52fb1d48dca048658a72aa55b3139c85"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "3582fd9d37e39dca9e821f297d74b18b"
server_key_iv: "7cfe8940c7fa518e608d896a723ceb76"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.902565 25660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:26:57.903227 25675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.903393 25660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:57.903496 25660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "52fb1d48dca048658a72aa55b3139c85"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "3582fd9d37e39dca9e821f297d74b18b"
server_key_iv: "7cfe8940c7fa518e608d896a723ceb76"
server_key_version: "encryptionkey@0"
I20260924 19:26:57.903568 25660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:57.934576 25660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:57.936583 25660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:57.936757 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:57.936996 25660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:57.937335 25660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:57.937389 25660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.937430 25660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:57.937457 25660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:57.943121 25660 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:37447
I20260924 19:26:57.943164 25788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:37447 every 8 connection(s)
I20260924 19:26:57.943526 25660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:26:57 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:26:57.949595 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25660
I20260924 19:26:57.949714 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:26:57.949941 19350 external_mini_cluster.cc:1428] Setting key 1fa8d7b71dc9b7e0b4a83503575e9ba1
I20260924 19:26:57.952917 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:57.944329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:41563 (local address 127.18.229.190:43627)
0924 19:26:57.944381 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:57.944384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:57.944574 (+ 190us) server_negotiation.cc:407] Connection header received
0924 19:26:57.945056 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:57.945057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:57.945068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:57.945121 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:57.946009 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.946486 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.947412 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.947589 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.947640 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:57.949852 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:57.949853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:57.949854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:57.949860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:57.951389 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:57.951910 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:57.951911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:57.951912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:57.951978 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:57.952295 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:57.952296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:57.952296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:57.952398 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:57.952452 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:57.952789 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 19:26:57.952855 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:26:57.953215 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:57.944305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:41563)
0924 19:26:57.944472 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:57.944508 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:57.944974 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:57.945177 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:57.945180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:57.945743 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:57.945932 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:57.945937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.946562 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.946563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:57.947340 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:57.947342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:57.947658 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:57.947659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:57.947755 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:57.948295 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:57.948304 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:57.949765 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:57.951495 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:57.951502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:57.951504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:57.951796 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:57.952050 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:57.952051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:57.952051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:57.952228 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:57.952465 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:57.952471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:57.952643 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 19:26:57.952765 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 19:26:57.952937 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260924 19:26:57.953708 25789 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43627
I20260924 19:26:57.953800 25789 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:57.953989 25789 heartbeater.cc:511] Master 127.18.229.190:43627 requested a full tablet report, sending...
I20260924 19:26:57.954394 25464 ts_manager.cc:194] Registered new tserver with Master: 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447)
I20260924 19:26:57.954849 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:41563
WARNING: no policy specified for oryx/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab.
Entry for principal oryx/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab.
I20260924 19:26:57.972890 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43627
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:58.051937 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:58.052176 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:58.052217 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:58.053376 25794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:58.053443 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:58.053490 25794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:58.054894 25794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:58.055152 25794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:58.055368 25794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.060200 25799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.060209 25800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.060209 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.060353 25794 server_base.cc:1034] running on GCE node
I20260924 19:26:58.060729 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.060994 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.062136 25794 hybrid_clock.cc:648] HybridClock initialized: now 1790278018062124 us; error 42 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.064415 25794 init.cc:377] Logged in from keytab as oryx/127.18.229.131@KRBTEST.COM (short username oryx)
I20260924 19:26:58.064806 25794 webserver.cc:460] Webserver started at http://127.18.229.131:34325/ using document root <none> and password file <none>
I20260924 19:26:58.065023 25794 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.065091 25794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.065245 25794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:58.066217 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "2ac4ea199a924848bcc47ca762894dd1"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8ec7cfe972961249800b6d62989fea03"
server_key_iv: "f081054bac10cbc3bcf9861905cdd7ea"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.066574 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "2ac4ea199a924848bcc47ca762894dd1"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8ec7cfe972961249800b6d62989fea03"
server_key_iv: "f081054bac10cbc3bcf9861905cdd7ea"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.067868 25794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:26:58.068621 25809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.068799 25794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.068892 25794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "2ac4ea199a924848bcc47ca762894dd1"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8ec7cfe972961249800b6d62989fea03"
server_key_iv: "f081054bac10cbc3bcf9861905cdd7ea"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.068996 25794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.098004 25794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.099813 25794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:58.099969 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.100253 25794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:58.100574 25794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:58.100605 25794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.100628 25794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:58.100672 25794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.106271 25794 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:39447
I20260924 19:26:58.106354 25922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:39447 every 8 connection(s)
I20260924 19:26:58.106714 25794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:58.107613 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25794
I20260924 19:26:58.107708 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:26:58.108031 19350 external_mini_cluster.cc:1428] Setting key a4ede5c358bc3863aa214748b2b5c029
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:26:58.115964 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.107443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:55643 (local address 127.18.229.190:43627)
0924 19:26:58.107502 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.107505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.107850 (+ 345us) server_negotiation.cc:407] Connection header received
0924 19:26:58.108580 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.108581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.108593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.108642 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.109615 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.110107 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.111142 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.111355 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.111438 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.113432 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.113433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.113433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.113440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.114635 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.115020 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.115021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.115021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.115085 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.115327 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.115328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.115328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.115435 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.115510 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.115764 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.115836 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:26:58.116196 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.107452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.131:55643)
0924 19:26:58.107609 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.107648 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.108447 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.108712 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.108716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.109318 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.109499 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.109505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.110202 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.110203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.111029 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.111031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.111445 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.111447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.111574 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.111984 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.111991 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.113329 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.114726 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.114732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.114733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.114933 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.115151 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.115151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.115152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.115251 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.115498 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.115504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.115636 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.115743 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.115902 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260924 19:26:58.116613 25923 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43627
I20260924 19:26:58.116712 25923 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.116892 25923 heartbeater.cc:511] Master 127.18.229.190:43627 requested a full tablet report, sending...
I20260924 19:26:58.117242 25464 ts_manager.cc:194] Registered new tserver with Master: 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447)
I20260924 19:26:58.117619 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:55643
I20260924 19:26:58.119618 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.229.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.189.keytab.
Entry for principal oryx/127.18.229.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.189.keytab.
I20260924 19:26:58.143213 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:46649
--webserver_interface=127.18.229.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.189.keytab
--principal=oryx/127.18.229.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43627,127.18.229.189:46649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:58.223893 25928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:58.224092 25928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:58.224128 25928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:58.225212 25928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:58.225248 25928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:58.225262 25928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:58.225275 25928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:58.225287 25928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:58.226679 25928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43627,127.18.229.189:46649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.189.keytab
--principal=oryx/127.18.229.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.229.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
W20260924 19:26:58.226774 25928 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.229.190:43627,127.18.229.189:46649'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 19:26:58.226881 25928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:58.227113 25928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.231917 25934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.232012 25936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.231933 25933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.232084 25928 server_base.cc:1034] running on GCE node
I20260924 19:26:58.232414 25928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.232748 25928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.233894 25928 hybrid_clock.cc:648] HybridClock initialized: now 1790278018233875 us; error 38 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.236330 25928 init.cc:377] Logged in from keytab as oryx/127.18.229.189@KRBTEST.COM (short username oryx)
I20260924 19:26:58.236783 25928 webserver.cc:460] Webserver started at http://127.18.229.189:44481/ using document root <none> and password file <none>
I20260924 19:26:58.237025 25928 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.237068 25928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.237233 25928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:58.238271 25928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/instance:
uuid: "80f4a776d1e14bbbb3d1b06f9155258b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8683e2d7e48ff13b0667fbf020995131"
server_key_iv: "24127f3955c2ae2dc4256f252f9b9959"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.238642 25928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal/instance:
uuid: "80f4a776d1e14bbbb3d1b06f9155258b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8683e2d7e48ff13b0667fbf020995131"
server_key_iv: "24127f3955c2ae2dc4256f252f9b9959"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.240017 25928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 19:26:58.240768 25943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.240935 25928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:58.241019 25928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
uuid: "80f4a776d1e14bbbb3d1b06f9155258b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8683e2d7e48ff13b0667fbf020995131"
server_key_iv: "24127f3955c2ae2dc4256f252f9b9959"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.241071 25928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.255451 25928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.257817 25928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:58.257927 25928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.261634 25928 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.189:46649
I20260924 19:26:58.261667 25995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.189:46649 every 8 connection(s)
I20260924 19:26:58.262034 25928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
I20260924 19:26:58.262993 25996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:58.264547 25996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:26:58.267644 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 25928
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.189@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
I20260924 19:26:58.274336 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.265237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.18.229.190:43627)
0924 19:26:58.265298 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.265300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.265500 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:26:58.266002 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.266004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.266017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.266067 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.266651 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.267142 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.268418 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.268713 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.268785 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.271104 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.271105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.271106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.271115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.272807 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.273255 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.273256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.273256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.273325 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.273553 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.273554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.273554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.273749 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.273805 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.274126 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.274222 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:26:58.274576 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.265217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:49216)
0924 19:26:58.265387 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.265419 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.265891 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.266137 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.266140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.266372 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.266550 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.266554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.267254 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.267255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.268330 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.268332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.268800 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.268801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.268894 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.269413 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.269423 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.271018 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.272891 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.272899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.272901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.273174 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.273401 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.273401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.273402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.273492 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.273818 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.273824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.273980 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.274092 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.274262 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260924 19:26:58.275244 25996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:26:58.277190 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.268150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.18.229.189:46649)
0924 19:26:58.268308 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.268313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.268357 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:26:58.268442 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.268445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.268472 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.268528 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.269192 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.270258 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.271137 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.271443 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.271524 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.274110 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.274117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.274117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.274124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.276009 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.276420 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.276422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.276423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.276480 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.276666 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.276667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.276667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.276821 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.276978 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.277031 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.277122 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.189@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
I20260924 19:26:58.283110 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.275535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.0.0.1:51236)
0924 19:26:58.275564 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.275574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.275611 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.275849 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.275849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.275991 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.276059 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.276061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.276967 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.276968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.277651 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.277654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.278049 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.278050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.278098 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.278486 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.278490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.279567 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.282009 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.282010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.282011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.282294 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.282495 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.282496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.282496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.282562 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.282807 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.282808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.282844 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.282911 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.283001 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 19:26:58.283442 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.275567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.18.229.189:46649)
0924 19:26:58.275728 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.275730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.275739 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:58.275760 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.275761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.275772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.275848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.276146 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.276896 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.277750 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.277987 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.278041 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.279636 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.279638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.279638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.279643 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.281932 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.282391 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.282392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.282393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.282447 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.282638 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.282639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.282639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.282740 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.282815 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.282938 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.283051 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260924 19:26:58.285012 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Bootstrap starting.
I20260924 19:26:58.285777 25996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:58.286080 25996 log.cc:824] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:58.286929 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: No bootstrap required, opened a new log
I20260924 19:26:58.288985 25996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } }
I20260924 19:26:58.289156 25996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:58.289206 25996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f4a776d1e14bbbb3d1b06f9155258b, State: Initialized, Role: FOLLOWER
I20260924 19:26:58.289322 25996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } }
I20260924 19:26:58.289562 26003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } } }
I20260924 19:26:58.289637 26003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:26:58.289616 25996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:26:58.291203 26014 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:58.291257 26014 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:58.292801 25928 master_runner.cc:428] Detected that this master 80f4a776d1e14bbbb3d1b06f9155258b is joining an existing cluster
I20260924 19:26:58.292847 25928 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.229.189:46649
I20260924 19:26:58.300210 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.293959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.0.0.1:51240)
0924 19:26:58.294124 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.294132 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.294178 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.294296 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.294297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.294459 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.294559 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.294561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.295225 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.295226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.295846 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.295848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.296356 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.296357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.296414 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.296867 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.296872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.297362 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.299092 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.299094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.299260 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.299467 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.299468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.299514 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.299807 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.299808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.299853 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.299940 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.300044 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":54,"threads_started":1}
I20260924 19:26:58.300217 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.294061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.18.229.189:46649)
0924 19:26:58.294111 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.294112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.294166 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:26:58.294229 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.294230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.294239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.294292 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.294625 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.295151 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.295935 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.296260 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.296343 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.297453 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.297455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.297455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.297461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.298995 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299350 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.299351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.299351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.299402 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299587 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.299588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.299588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.299729 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.299806 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.299979 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.300127 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":31}
I20260924 19:26:58.300593 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.293809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.18.229.190:43627)
0924 19:26:58.293854 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.293856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.293999 (+ 143us) server_negotiation.cc:407] Connection header received
0924 19:26:58.294025 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.294025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.294038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.294105 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.294506 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.295270 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.296360 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.296661 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.296715 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.297739 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.297741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.297742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.297750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.299338 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299678 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.299679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.299679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.299747 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299980 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.299981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.299981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.300106 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.300175 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.300392 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.300480 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:26:58.300915 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.293745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:49220)
0924 19:26:58.293895 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.293905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.293960 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.294148 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.294149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.294334 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.294410 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.294412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.295368 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.295369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.296262 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.296265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.296790 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.296791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.296864 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.297261 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.297265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.297625 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.299414 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.299415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.299609 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.299833 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.299833 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.299834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.299905 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.300224 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.300224 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.300265 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.300672 (+ 407us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.300797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.190@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
I20260924 19:26:58.310197 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.303055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.18.229.189:46649)
0924 19:26:58.303091 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.303093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.303188 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:26:58.303269 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.303270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.303276 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.303316 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.303783 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.304268 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.305345 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.305506 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.305570 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.307613 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.307614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.307614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.307627 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.309011 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.309413 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.309414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.309414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.309470 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.309669 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.309670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.309670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.309782 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.309856 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.310030 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.310108 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:26:58.310243 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.302956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.0.0.1:51248)
0924 19:26:58.303078 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.303122 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.303196 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.303331 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.303334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.303649 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.303712 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.303715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.304337 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.304338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.305268 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.305271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.305575 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.305576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.305657 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.306050 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.306055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.307516 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.309112 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.309119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.309121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.309325 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.309536 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.309537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.309538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.309605 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.309849 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.309856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.309906 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.310006 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.310118 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260924 19:26:58.310890 25464 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.229.189:46649: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b"
member_type: NON_VOTER
last_known_addr {
host: "127.18.229.189"
port: 46649
}
attrs {
promote: true
}
}
dest_uuid: "261791e4f0ef4077b728544ae9a8b4af"
cas_config_opid_index: -1
I20260924 19:26:58.311210 25464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } }
I20260924 19:26:58.311719 26024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 80f4a776d1e14bbbb3d1b06f9155258b (127.18.229.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } }
I20260924 19:26:58.311949 25970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:58.312748 25970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Refusing update from remote peer 261791e4f0ef4077b728544ae9a8b4af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 19:26:58.312785 26024 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.229.189:46649
I20260924 19:26:58.312862 26024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } }
I20260924 19:26:58.312917 26024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:58.313011 25928 master.cc:537] Master@127.18.229.189:46649 shutting down...
I20260924 19:26:58.313251 26023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:26:58.314591 25450 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 19:26:58.315294 25450 consensus_peers.cc:487] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af -> Peer 80f4a776d1e14bbbb3d1b06f9155258b (127.18.229.189:46649): Couldn't send request to peer 80f4a776d1e14bbbb3d1b06f9155258b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:26:58.315725 25928 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:58.315811 25928 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:58.315861 25928 tablet_replica.cc:330] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: stopping tablet replica
I20260924 19:26:58.328826 25928 master.cc:552] Master@127.18.229.189:46649 shutdown complete.
I20260924 19:26:58.328899 25928 master_runner.cc:305] Clearing existing system tablet
I20260924 19:26:58.329034 25928 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.329093 25928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.329728 25928 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:58.330178 26028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.330300 25928 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.330355 25928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.330377 25928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
uuid: "80f4a776d1e14bbbb3d1b06f9155258b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8683e2d7e48ff13b0667fbf020995131"
server_key_iv: "24127f3955c2ae2dc4256f252f9b9959"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.331044 25928 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:58.332399 25928 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:26:58.332455 25928 log.cc:1196] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 19:26:58.332670 25928 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Deleting consensus metadata
I20260924 19:26:58.332795 25928 master_runner.cc:315] Copying system tablet from 127.18.229.190:43627
I20260924 19:26:58.333423 25928 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: tablet copy: Beginning tablet copy session from remote peer at address 127.18.229.190:43627
I20260924 19:26:58.339728 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.333715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:49230)
0924 19:26:58.333864 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.333874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.333946 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.334088 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.334088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.334234 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.334287 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.334288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.335232 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.335233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.335768 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.335770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.336285 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.336286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.336335 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.336828 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.336833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.337125 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.338829 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.338830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.338830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.338990 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.339184 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.339184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.339184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.339236 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.339500 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.339500 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.339534 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.339585 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.339657 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260924 19:26:58.339783 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.333801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.18.229.190:43627)
0924 19:26:58.333861 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.333863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.333925 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:26:58.334014 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.334015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.334026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.334086 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.334354 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.335132 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.335846 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.336212 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.336286 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.337206 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.337207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.337208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.337214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.338749 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.339059 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.339060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.339061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.339121 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.339294 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.339295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.339296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.339429 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.339507 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.339608 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.339682 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:26:58.340509 25484 tablet_copy_service.cc:140] P 261791e4f0ef4077b728544ae9a8b4af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 80f4a776d1e14bbbb3d1b06f9155258b ({username='oryx', principal='oryx/127.18.229.189@KRBTEST.COM'} at 127.0.0.1:49230)
I20260924 19:26:58.340592 25484 tablet_copy_service.cc:161] P 261791e4f0ef4077b728544ae9a8b4af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 80f4a776d1e14bbbb3d1b06f9155258b at {username='oryx', principal='oryx/127.18.229.189@KRBTEST.COM'} at 127.0.0.1:49230: session id = 80f4a776d1e14bbbb3d1b06f9155258b-00000000000000000000000000000000
I20260924 19:26:58.341236 25484 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:26:58.341861 25928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:26:58.342854 25928 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: tablet copy: Starting download of 0 data blocks...
I20260924 19:26:58.343010 25928 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: tablet copy: Starting download of 1 WAL segments...
I20260924 19:26:58.343876 25928 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:26:58.344652 25484 tablet_copy_service.cc:342] P 261791e4f0ef4077b728544ae9a8b4af: Request end of tablet copy session 80f4a776d1e14bbbb3d1b06f9155258b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.229.189@KRBTEST.COM'} at 127.0.0.1:49230
I20260924 19:26:58.344717 25484 tablet_copy_service.cc:433] P 261791e4f0ef4077b728544ae9a8b4af: ending tablet copy session 80f4a776d1e14bbbb3d1b06f9155258b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 80f4a776d1e14bbbb3d1b06f9155258b
I20260924 19:26:58.346942 25928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.348322 26036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.348333 25928 server_base.cc:1034] running on GCE node
W20260924 19:26:58.348322 26035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.348340 26038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.348608 25928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.348769 25928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.349893 25928 hybrid_clock.cc:648] HybridClock initialized: now 1790278018349886 us; error 25 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.351202 25928 init.cc:377] Logged in from keytab as oryx/127.18.229.189@KRBTEST.COM (short username oryx)
I20260924 19:26:58.351468 25928 webserver.cc:460] Webserver started at http://127.18.229.189:35233/ using document root <none> and password file <none>
I20260924 19:26:58.351579 25928 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.351605 25928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.352221 25928 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.352734 26045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.352856 25928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.352901 25928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
uuid: "80f4a776d1e14bbbb3d1b06f9155258b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8683e2d7e48ff13b0667fbf020995131"
server_key_iv: "24127f3955c2ae2dc4256f252f9b9959"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.352957 25928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.375257 25928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.376928 25928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.380024 25928 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.189:46649
I20260924 19:26:58.380041 26097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.189:46649 every 8 connection(s)
I20260924 19:26:58.380389 25928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
I20260924 19:26:58.381310 26098 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:26:58.381886 26098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Bootstrap starting.
I20260924 19:26:58.384800 26098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:26:58.385044 26098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Bootstrap complete.
I20260924 19:26:58.385241 26098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } }
I20260924 19:26:58.385329 26098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80f4a776d1e14bbbb3d1b06f9155258b, State: Initialized, Role: LEARNER
I20260924 19:26:58.385367 26098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } }
I20260924 19:26:58.385550 26098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:58.385552 26101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } }
I20260924 19:26:58.385663 26101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: LEARNER
I20260924 19:26:58.387085 26112 catalog_manager.cc:1259] Loaded cluster ID: 9e3e6509a910435389bb4c6ff093c579
I20260924 19:26:58.387672 26112 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: loading cluster ID for follower catalog manager: success
I20260924 19:26:58.388489 26112 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: acquiring CA information for follower catalog manager: success
I20260924 19:26:58.388748 26112 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:26:58.435319 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.429323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.18.229.189:46649)
0924 19:26:58.429483 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.429485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.429491 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:26:58.429514 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.429514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.429523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.429603 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.430144 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.430793 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.431521 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.431729 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.431819 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.433009 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.433010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.433011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.433018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.434179 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.434536 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.434537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.434537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.434599 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.434801 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.434801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.434802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.434916 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.435022 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.435117 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.435210 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.18.229.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.188.keytab.
Entry for principal oryx/127.18.229.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.188.keytab.
I20260924 19:26:58.457008 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:43229
--webserver_interface=127.18.229.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.188.keytab
--principal=oryx/127.18.229.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:26:58.535830 26116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:58.536003 26116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:58.536023 26116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:58.537192 26116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:26:58.537232 26116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:58.537246 26116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:58.537284 26116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:26:58.537310 26116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:26:58.538699 26116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.188.keytab
--principal=oryx/127.18.229.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.229.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:58.538895 26116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:58.539088 26116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.543747 26124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.543756 26122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.543754 26116 server_base.cc:1034] running on GCE node
W20260924 19:26:58.543943 26121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.544226 26116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.544457 26116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.545586 26116 hybrid_clock.cc:648] HybridClock initialized: now 1790278018545575 us; error 28 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.547827 26116 init.cc:377] Logged in from keytab as oryx/127.18.229.188@KRBTEST.COM (short username oryx)
I20260924 19:26:58.548255 26116 webserver.cc:460] Webserver started at http://127.18.229.188:36901/ using document root <none> and password file <none>
I20260924 19:26:58.548444 26116 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.548481 26116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.548563 26116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:26:58.549480 26116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/instance:
uuid: "ca97866e1f5b4d03978f39b568fb221b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "2ef7e91d33b06a5fe2f9dd78b20b18d8"
server_key_iv: "299289870a2bb59f50d0bd339e81fc9e"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.549809 26116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal/instance:
uuid: "ca97866e1f5b4d03978f39b568fb221b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "2ef7e91d33b06a5fe2f9dd78b20b18d8"
server_key_iv: "299289870a2bb59f50d0bd339e81fc9e"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.550994 26116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:26:58.551702 26131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.551847 26116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:58.551928 26116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
uuid: "ca97866e1f5b4d03978f39b568fb221b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "2ef7e91d33b06a5fe2f9dd78b20b18d8"
server_key_iv: "299289870a2bb59f50d0bd339e81fc9e"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.551978 26116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.567245 26116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.569518 26116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:58.569620 26116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.573398 26116 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.188:43229
I20260924 19:26:58.573443 26183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.188:43229 every 8 connection(s)
I20260924 19:26:58.573753 26116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
I20260924 19:26:58.574682 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:58.576243 26184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.188@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:26:58.581298 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 26116
I20260924 19:26:58.584555 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.577024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.18.229.190:43627)
0924 19:26:58.577073 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.577075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.577118 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:26:58.577544 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.577545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.577554 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.577606 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.578138 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.578608 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.579470 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.579645 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.579701 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.581706 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.581708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.581708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.581716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.583158 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.583593 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.583595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.583595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.583663 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.583887 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.583888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.583888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.584018 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.584083 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.584392 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.584468 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:26:58.584586 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.576864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:49236)
0924 19:26:58.577021 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.577053 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.577460 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.577670 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.577673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.577892 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.578068 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.578071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.578677 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.578677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.579399 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.579401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.579729 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.579729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.579829 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.580261 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.580268 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.581595 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.583259 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.583265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.583267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.583514 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.583733 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.583733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.583733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.583824 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.584079 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.584082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.584256 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.584339 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.584445 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260924 19:26:58.585325 26184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.229.188@KRBTEST.COM
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.188@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
I20260924 19:26:58.590241 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.581795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.18.229.188:43229)
0924 19:26:58.581943 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.581948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.581967 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:26:58.581986 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.581988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.582007 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.582087 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.582632 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.583285 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.584209 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.584626 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.584724 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.587339 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.587344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.587345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.587351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.589067 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.589424 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.589426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.589426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.589479 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.589669 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.589670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.589671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.589821 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.589954 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.590062 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.590130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260924 19:26:58.592022 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.585607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.0.0.1:51278)
0924 19:26:58.585720 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.585730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.585783 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.585976 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.585977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.586084 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.586171 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.586173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.586931 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.586932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.587655 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.587657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.588191 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.588192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.588260 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.588531 (+ 271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.588536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.589578 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.591321 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.591321 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.591322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.591467 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.591610 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.591611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.591611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.591655 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.591832 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.591833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.591863 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.591911 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.591971 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 19:26:58.592036 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.585671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.18.229.189:46649)
0924 19:26:58.585719 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.585720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.585769 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:26:58.585846 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.585847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.585857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.585939 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.586282 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.586826 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.587755 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.588054 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.588091 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.589635 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.589636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.589636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.589644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.591259 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.591522 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.591523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.591523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.591562 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.591704 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.591705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.591705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.591785 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.591827 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.591926 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.591981 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:26:58.592401 26184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.188@KRBTEST.COM for oryx/127.18.229.188@KRBTEST.COM
I20260924 19:26:58.597909 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.592654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:47572)
0924 19:26:58.592711 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.592721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.592759 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.592873 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.592873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.592971 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.592994 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.592995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.593513 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.593513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.593951 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.593952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.594244 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.594261 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.594313 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.594620 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.594624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.595730 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.597152 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.597153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.597153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.597305 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.597459 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.597459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.597460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.597533 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.597722 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.597722 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.597750 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.597798 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.597860 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 19:26:58.597966 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.592712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.18.229.188:43229)
0924 19:26:58.592752 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.592754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.592761 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:26:58.592817 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.592817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.592824 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.592866 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.593041 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.593458 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.594014 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.594185 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.594258 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.595796 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.595797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.595797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.595802 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.597093 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.597364 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.597365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.597365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.597407 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.597594 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.597595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.597595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.597667 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.597714 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.597810 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.597902 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":19}
I20260924 19:26:58.599046 26184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Bootstrap starting.
I20260924 19:26:58.599809 26184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:58.600060 26184 log.cc:824] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:58.600745 26184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: No bootstrap required, opened a new log
I20260924 19:26:58.602084 26184 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } }
I20260924 19:26:58.602219 26184 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:58.602267 26184 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca97866e1f5b4d03978f39b568fb221b, State: Initialized, Role: FOLLOWER
I20260924 19:26:58.602404 26184 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } }
I20260924 19:26:58.602691 26184 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:58.602681 26189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } } }
I20260924 19:26:58.602826 26189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:26:58.604251 26200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:26:58.604414 26200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:26:58.606407 26116 master_runner.cc:428] Detected that this master ca97866e1f5b4d03978f39b568fb221b is joining an existing cluster
I20260924 19:26:58.606487 26116 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.229.188:43229
I20260924 19:26:58.613736 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.607453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.18.229.188:43229)
0924 19:26:58.607486 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.607487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.607759 (+ 272us) server_negotiation.cc:407] Connection header received
0924 19:26:58.607835 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.607836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.607843 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.607867 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.608358 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.609050 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610059 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.610315 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610352 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.611431 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.611432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.611432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.611436 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.612729 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.613099 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.613100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.613100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.613145 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.613333 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.613333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.613333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.613417 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.613463 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.613618 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.613677 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:26:58.614157 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.607444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:47588)
0924 19:26:58.607716 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.607724 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.607776 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.607924 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.607925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.608206 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.608276 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.608278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.609133 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.609133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.609975 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.609977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.610425 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.610474 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.610929 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.610934 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.611352 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.612807 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.612808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.612809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.613022 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.613205 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.613206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.613206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.613270 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.613478 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.613479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.613530 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.613602 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.613748 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":50,"thread_start_us":81,"threads_started":1}
I20260924 19:26:58.636685 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.607681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.18.229.189:46649)
0924 19:26:58.607719 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.607721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.607830 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:26:58.607928 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.607929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.607937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.607968 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.608442 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.609155 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610309 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.610595 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610650 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.633577 (+ 22927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.633579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.633580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.633588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.635437 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.635816 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.635818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.635818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.635876 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.636101 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.636103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.636103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.636249 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.636328 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.636507 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.636589 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:26:58.636711 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.607639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.0.0.1:51288)
0924 19:26:58.607772 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.607783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.607856 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.608031 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.608032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.608311 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.608358 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.608360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.609254 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.609254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.610223 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.610225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.610736 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.610801 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.632931 (+ 22130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.632935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.633459 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.635520 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.635521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.635521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.635741 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.635975 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.635975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.635976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.636025 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.636322 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.636323 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.636377 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.636482 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.636589 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":58,"thread_start_us":66,"threads_started":1}
I20260924 19:26:58.637166 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:49252)
0924 19:26:58.607572 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.607581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.607631 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.607969 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.607970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.608246 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.608313 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.608315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.609254 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.609254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.610213 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.610215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.610737 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.610801 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.632854 (+ 22053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.632860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.633459 (+ 599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.635419 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.635420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.635421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.635690 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.635926 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.635927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.635927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.635987 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.636860 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.636861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.636903 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.636964 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.637065 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":58,"thread_start_us":86,"threads_started":1}
I20260924 19:26:58.637631 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.607795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.18.229.190:43627)
0924 19:26:58.607834 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.607836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.607843 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:26:58.607865 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.607866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.607876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.607926 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.608411 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.609156 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610310 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.610595 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.610650 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.633610 (+ 22960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.633612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.633613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.633620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.635315 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.635770 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.635771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.635772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.635852 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.636059 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.636060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.636060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.636230 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.636335 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.637449 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.637535 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:26:58.641691 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25526
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278017, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.190@KRBTEST.COM for oryx/127.18.229.188@KRBTEST.COM
I20260924 19:26:58.646704 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.639533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:47592)
0924 19:26:58.639583 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.639592 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.639626 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.639780 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.639781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.639941 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.639994 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.639996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.640650 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.640651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.641484 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.641487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.641985 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.641986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.642067 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.642489 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.642493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.643936 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.645697 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.645698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.645699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.645897 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.646095 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.646096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.646096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.646151 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.646406 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.646407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.646449 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.646508 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.646605 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:26:58.647025 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.639604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.18.229.188:43229)
0924 19:26:58.639653 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.639654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.639662 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:58.639706 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.639707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.639715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.639761 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.640071 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.640574 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.641586 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.641876 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.641938 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.644029 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.644031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.644031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.644037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.645605 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.645970 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.645971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.645971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.646019 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.646213 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.646213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.646214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.646329 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.646382 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.646870 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.646941 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:26:58.647578 25464 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.229.188:43229: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b"
member_type: NON_VOTER
last_known_addr {
host: "127.18.229.188"
port: 43229
}
attrs {
promote: true
}
}
dest_uuid: "261791e4f0ef4077b728544ae9a8b4af"
cas_config_opid_index: 5
I20260924 19:26:58.647841 25464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } }
I20260924 19:26:58.648222 26023 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ca97866e1f5b4d03978f39b568fb221b (127.18.229.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.648785 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:37889
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:43627
--webserver_port=33841
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:26:58.649128 26023 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.229.188:43229
I20260924 19:26:58.649232 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.649318 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:58.649762 26116 master.cc:537] Master@127.18.229.188:43229 shutting down...
I20260924 19:26:58.649904 26158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:58.651055 26158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 FOLLOWER]: Refusing update from remote peer 261791e4f0ef4077b728544ae9a8b4af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:26:58.651414 26024 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:26:58.652252 25448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af -> Peer ca97866e1f5b4d03978f39b568fb221b (127.18.229.188:43229): Couldn't send request to peer ca97866e1f5b4d03978f39b568fb221b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:26:58.653271 26116 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:58.653360 26116 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:58.653411 26116 tablet_replica.cc:330] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: stopping tablet replica
I20260924 19:26:58.655854 26116 master.cc:552] Master@127.18.229.188:43229 shutdown complete.
I20260924 19:26:58.655898 26116 master_runner.cc:305] Clearing existing system tablet
I20260924 19:26:58.656023 26116 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.656076 26116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.656279 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.649517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.0.0.1:51304)
0924 19:26:58.650986 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.650996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.651044 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.651260 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.651261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.651263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:58.651302 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.651304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.653158 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.653159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.655018 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.655021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.655953 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.655954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.656026 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.656051 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.656102 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.656178 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453}
I20260924 19:26:58.656548 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.649567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.18.229.189:46649)
0924 19:26:58.650954 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.650956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.651093 (+ 137us) server_negotiation.cc:407] Connection header received
0924 19:26:58.651118 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.651119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.651140 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.651202 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:58.652339 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.653061 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.655108 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.655776 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.655849 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.655920 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.656422 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.656463 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362}
I20260924 19:26:58.656780 26071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 LEARNER]: Refusing update from remote peer 261791e4f0ef4077b728544ae9a8b4af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:26:58.657040 26024 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 19:26:58.657598 26071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ca97866e1f5b4d03978f39b568fb221b (127.18.229.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.658818 26024 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: attempting to promote NON_VOTER 80f4a776d1e14bbbb3d1b06f9155258b to VOTER
I20260924 19:26:58.658872 26101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261791e4f0ef4077b728544ae9a8b4af. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.658972 26101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: LEARNER
I20260924 19:26:58.659018 26101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: NON_VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: true } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.659001 26024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } }
I20260924 19:26:58.659080 26101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: LEARNER
I20260924 19:26:58.659476 26071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 LEARNER]: Refusing update from remote peer 261791e4f0ef4077b728544ae9a8b4af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 19:26:58.659547 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.659437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:47596)
0924 19:26:58.659478 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.659516 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:26:58.659633 26023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 19:26:58.660022 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.659931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:47606)
0924 19:26:58.659969 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.659999 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 19:26:58.660097 25448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af -> Peer ca97866e1f5b4d03978f39b568fb221b (127.18.229.188:43229): Couldn't send request to peer ca97866e1f5b4d03978f39b568fb221b. Status: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:26:58.660382 26023 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 80f4a776d1e14bbbb3d1b06f9155258b (127.18.229.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.661206 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.661281 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:58.661327 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.661374 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:58.661607 26071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 80f4a776d1e14bbbb3d1b06f9155258b (127.18.229.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.662586 26101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.662683 26101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:26:58.668390 26116 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.000s sys 0.000s
I20260924 19:26:58.668937 26214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.669039 26116 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.669080 26116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:58.669101 26116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
uuid: "ca97866e1f5b4d03978f39b568fb221b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "2ef7e91d33b06a5fe2f9dd78b20b18d8"
server_key_iv: "299289870a2bb59f50d0bd339e81fc9e"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.669824 26116 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:58.671160 26116 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:26:58.671233 26116 log.cc:1196] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 19:26:58.671425 26116 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Deleting consensus metadata
I20260924 19:26:58.671535 26116 master_runner.cc:315] Copying system tablet from 127.18.229.190:43627
I20260924 19:26:58.672097 26116 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: tablet copy: Beginning tablet copy session from remote peer at address 127.18.229.190:43627
I20260924 19:26:58.679162 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.672392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:49256)
0924 19:26:58.672534 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.672556 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.672615 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.672760 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.672761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.672960 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.673010 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.673012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.673911 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.673912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.674709 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.674711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.675215 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.675216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.675277 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.675786 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.675792 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.676210 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.678024 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.678025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.678026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.678273 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.678514 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.678515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.678515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.678578 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.678862 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.678863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.678907 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.678978 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.679073 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260924 19:26:58.679179 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.672603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.18.229.190:43627)
0924 19:26:58.672639 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.672641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.672650 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:58.672674 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.672674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.672685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.672745 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.673105 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.673819 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.674802 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.675120 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.675212 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.676295 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.676297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.676297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.676305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.677938 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.678350 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.678351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.678351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.678425 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.678648 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.678649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.678649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.678789 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.678859 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.679018 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.679098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:26:58.679677 25484 tablet_copy_service.cc:140] P 261791e4f0ef4077b728544ae9a8b4af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ca97866e1f5b4d03978f39b568fb221b ({username='oryx', principal='oryx/127.18.229.188@KRBTEST.COM'} at 127.0.0.1:49256)
I20260924 19:26:58.679728 25484 tablet_copy_service.cc:161] P 261791e4f0ef4077b728544ae9a8b4af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ca97866e1f5b4d03978f39b568fb221b at {username='oryx', principal='oryx/127.18.229.188@KRBTEST.COM'} at 127.0.0.1:49256: session id = ca97866e1f5b4d03978f39b568fb221b-00000000000000000000000000000000
I20260924 19:26:58.680321 25484 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:26:58.680791 26116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:26:58.682073 26116 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: tablet copy: Starting download of 0 data blocks...
I20260924 19:26:58.682221 26116 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: tablet copy: Starting download of 1 WAL segments...
I20260924 19:26:58.682894 26116 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:26:58.683741 25484 tablet_copy_service.cc:342] P 261791e4f0ef4077b728544ae9a8b4af: Request end of tablet copy session ca97866e1f5b4d03978f39b568fb221b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.229.188@KRBTEST.COM'} at 127.0.0.1:49256
I20260924 19:26:58.683797 25484 tablet_copy_service.cc:433] P 261791e4f0ef4077b728544ae9a8b4af: ending tablet copy session ca97866e1f5b4d03978f39b568fb221b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ca97866e1f5b4d03978f39b568fb221b
I20260924 19:26:58.686326 26116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.687546 26224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.687590 26221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.687834 26222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.693030 26116 server_base.cc:1034] running on GCE node
I20260924 19:26:58.693156 26116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.693318 26116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.694425 26116 hybrid_clock.cc:648] HybridClock initialized: now 1790278018694416 us; error 30 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.695955 26116 init.cc:377] Logged in from keytab as oryx/127.18.229.188@KRBTEST.COM (short username oryx)
I20260924 19:26:58.696336 26116 webserver.cc:460] Webserver started at http://127.18.229.188:42735/ using document root <none> and password file <none>
I20260924 19:26:58.696457 26116 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.696491 26116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.697235 26116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:58.697926 26231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.698112 26116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:58.698175 26116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
uuid: "ca97866e1f5b4d03978f39b568fb221b"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "2ef7e91d33b06a5fe2f9dd78b20b18d8"
server_key_iv: "299289870a2bb59f50d0bd339e81fc9e"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.698257 26116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.712213 26116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.714033 26116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.716936 26116 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.188:43229
I20260924 19:26:58.717259 26116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
I20260924 19:26:58.720243 26283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.188:43229 every 8 connection(s)
I20260924 19:26:58.720961 26284 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:26:58.721438 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Bootstrap starting.
I20260924 19:26:58.724334 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:26:58.724540 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: Bootstrap complete.
I20260924 19:26:58.724735 26284 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } }
I20260924 19:26:58.724823 26284 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca97866e1f5b4d03978f39b568fb221b, State: Initialized, Role: LEARNER
I20260924 19:26:58.724856 26284 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } }
I20260924 19:26:58.725039 26284 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:26:58.725028 26287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:58.725276 26287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: This master's current role is: LEARNER
I20260924 19:26:58.726526 26298 catalog_manager.cc:1259] Loaded cluster ID: 9e3e6509a910435389bb4c6ff093c579
I20260924 19:26:58.726634 26298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: loading cluster ID for follower catalog manager: success
I20260924 19:26:58.727788 26298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: acquiring CA information for follower catalog manager: success
I20260924 19:26:58.728089 26298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:26:58.751299 26209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:58.751510 26209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:58.751547 26209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:58.752828 26209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:58.752892 26209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:58.752934 26209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:26:58.754418 26209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:37889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=33841
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:58.754676 26209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:58.754908 26209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.760301 26306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.760308 26304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.760428 26303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.760308 26209 server_base.cc:1034] running on GCE node
I20260924 19:26:58.760628 26209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.760901 26209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.762060 26209 hybrid_clock.cc:648] HybridClock initialized: now 1790278018762043 us; error 31 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.764348 26209 init.cc:377] Logged in from keytab as oryx/127.18.229.129@KRBTEST.COM (short username oryx)
I20260924 19:26:58.764806 26209 webserver.cc:460] Webserver started at http://127.18.229.129:33841/ using document root <none> and password file <none>
I20260924 19:26:58.765057 26209 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.765134 26209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.766402 26209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:58.766959 26313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.767127 26209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:58.767222 26209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "f885e65c6bdc477a8baa2b67ba10ce08"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "627890260d093b9009721f01a572d062"
server_key_iv: "6d9ba8fb22c979156f53e857053186f9"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.767549 26209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.789079 26209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.790812 26209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:58.790967 26209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.791182 26209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:58.791476 26209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:58.791529 26209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.791569 26209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:58.791597 26209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.797544 26209 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:37889
I20260924 19:26:58.797583 26426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:37889 every 8 connection(s)
I20260924 19:26:58.797969 26209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.188@KRBTEST.COM
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
I20260924 19:26:58.806694 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 26209
I20260924 19:26:58.806859 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25660
I20260924 19:26:58.809404 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.798718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:52469 (local address 127.18.229.190:43627)
0924 19:26:58.798788 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.798790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.799002 (+ 212us) server_negotiation.cc:407] Connection header received
0924 19:26:58.800246 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.800247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.800258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.800290 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.801074 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.801848 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803150 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.803434 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803510 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.805779 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.805781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.805781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.805802 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.807724 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.808274 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.808276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.808276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.808335 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.808607 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.808608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.808609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.808748 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.808802 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.809155 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.809257 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:26:58.809767 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.799343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:35221 (local address 127.18.229.189:46649)
0924 19:26:58.799395 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.799397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.799407 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:58.800193 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.800195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.800209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.800256 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.801073 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.801893 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.802992 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.803264 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803327 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.806740 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.806741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.806742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.806751 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.808564 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.808953 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.808954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.808954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.809005 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.809214 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.809215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.809215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.809361 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.809406 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.809599 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.809673 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:26:58.809933 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.798705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:52469)
0924 19:26:58.798875 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.798909 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.800073 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.800362 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.800363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.800766 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.800963 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.800969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.801964 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.801964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.803071 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.803073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.803509 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.803575 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.804259 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.804264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.805633 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.807830 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.807836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.807838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.808186 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.808406 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.808407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.808407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.808534 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.808832 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.808837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.808993 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.809524 (+ 531us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.809698 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":60,"thread_start_us":80,"threads_started":1}
I20260924 19:26:58.809962 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.798981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.18.229.129:35221)
0924 19:26:58.799112 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.799122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.800097 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.800343 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.800347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.800736 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.800964 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.800970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.801998 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.801998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.802901 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.802903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.803345 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.803473 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.803940 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.803948 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.806660 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.808650 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.808651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.808652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.808873 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.809079 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.809080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.809080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.809137 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.809454 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.809455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.809498 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.809567 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.809789 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":144,"thread_start_us":86,"threads_started":1}
I20260924 19:26:58.810398 26429 heartbeater.cc:347] Connected to a master server at 127.18.229.189:46649
I20260924 19:26:58.810420 26428 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43627
I20260924 19:26:58.810482 26429 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.810483 26428 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.810619 26429 heartbeater.cc:511] Master 127.18.229.189:46649 requested a full tablet report, sending...
I20260924 19:26:58.810628 26428 heartbeater.cc:511] Master 127.18.229.190:43627 requested a full tablet report, sending...
I20260924 19:26:58.811040 25464 ts_manager.cc:194] Re-registered known tserver with Master: f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889)
I20260924 19:26:58.811074 26062 ts_manager.cc:194] Registered new tserver with Master: f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889)
I20260924 19:26:58.811566 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:52469
I20260924 19:26:58.813171 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:37447
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:43627
--webserver_port=37045
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:26:58.814893 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.799257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:45349 (local address 127.18.229.188:43229)
0924 19:26:58.799455 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.799457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.799466 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:58.800281 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.800282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.800295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.800370 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.801073 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.802070 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803167 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.803442 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803500 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.806214 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.806216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.806216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.806223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.813795 (+ 7572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.814111 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.814112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.814113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.814177 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.814377 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.814378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.814378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.814495 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.814572 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.814716 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.814795 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":47,"threads_started":1}
I20260924 19:26:58.815065 26433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.799187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.18.229.129:45349)
0924 19:26:58.799327 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.799336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.800096 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.800373 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.800374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.800778 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.800963 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.800969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.802151 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.802152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.803075 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.803076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.803552 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.803552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.803612 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.804038 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.804043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.806125 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.813881 (+ 7756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.813882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.813882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.814046 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.814248 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.814249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.814249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.814295 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.814564 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.814565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.814610 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.814685 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.814791 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260924 19:26:58.815474 26427 heartbeater.cc:347] Connected to a master server at 127.18.229.188:43229
I20260924 19:26:58.815521 26427 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.815595 26427 heartbeater.cc:511] Master 127.18.229.188:43229 requested a full tablet report, sending...
I20260924 19:26:58.815943 26248 ts_manager.cc:194] Registered new tserver with Master: f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889)
W20260924 19:26:58.895177 26435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:58.895390 26435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:58.895427 26435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:58.896700 26435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:58.896755 26435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:58.896800 26435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:26:58.898375 26435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:37447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=37045
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:58.898624 26435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:58.898845 26435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:58.903663 26440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:58.903751 26443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.903903 26435 server_base.cc:1034] running on GCE node
W20260924 19:26:58.903676 26441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:58.904229 26435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:58.904419 26435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:58.905565 26435 hybrid_clock.cc:648] HybridClock initialized: now 1790278018905553 us; error 27 us; skew 500 ppm
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:58.907992 26435 init.cc:377] Logged in from keytab as oryx/127.18.229.130@KRBTEST.COM (short username oryx)
I20260924 19:26:58.908450 26435 webserver.cc:460] Webserver started at http://127.18.229.130:37045/ using document root <none> and password file <none>
I20260924 19:26:58.908677 26435 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:58.908761 26435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:58.910043 26435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:26:58.910593 26450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.910753 26435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:26:58.910848 26435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "52fb1d48dca048658a72aa55b3139c85"
format_stamp: "Formatted at 2026-09-24 19:26:57 on dist-test-slave-r8cc"
server_key: "3582fd9d37e39dca9e821f297d74b18b"
server_key_iv: "7cfe8940c7fa518e608d896a723ceb76"
server_key_version: "encryptionkey@0"
I20260924 19:26:58.911180 26435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:58.927381 26435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:58.929162 26435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:58.929320 26435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:58.929534 26435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:58.929841 26435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:58.929898 26435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.929940 26435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:58.929967 26435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:58.935401 26435 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:37447
I20260924 19:26:58.935474 26563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:37447 every 8 connection(s)
I20260924 19:26:58.935822 26435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:26:58.938257 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 26435
I20260924 19:26:58.938418 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25794
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
Sep 24 19:26:58 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278018, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.188@KRBTEST.COM
I20260924 19:26:58.945474 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:39447
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:43627
--webserver_port=34325
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:26:58.948163 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.936866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:35059 (local address 127.18.229.189:46649)
0924 19:26:58.936911 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.936913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.937028 (+ 115us) server_negotiation.cc:407] Connection header received
0924 19:26:58.937889 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.937890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.937901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.937946 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.939244 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.940013 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941358 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.941699 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941739 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.944672 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.944674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.944675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.944683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.946515 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947004 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.947005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.947006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.947061 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947360 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.947361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.947361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.947506 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.947549 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.947937 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.948020 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:26:58.948798 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.936577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:36039 (local address 127.18.229.190:43627)
0924 19:26:58.936630 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.936632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.936990 (+ 358us) server_negotiation.cc:407] Connection header received
0924 19:26:58.937908 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.937910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.937923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.937956 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.938908 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.939711 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941319 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.941622 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941663 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.945007 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.945008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.945009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.945016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.947432 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947927 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.947928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.947928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.947989 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.948248 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.948249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.948250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.948396 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.948436 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.948628 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.948715 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:26:58.948892 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.936582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:45081 (local address 127.18.229.188:43229)
0924 19:26:58.936630 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:58.936632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:58.936824 (+ 192us) server_negotiation.cc:407] Connection header received
0924 19:26:58.937908 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:58.937909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:58.937925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:58.937956 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:58.939027 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.939744 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941318 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.941622 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941663 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.945884 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:58.945886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:58.945886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:58.945893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:58.947380 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947928 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.947929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.947929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.947989 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:58.948230 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:58.948231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:58.948232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:58.948357 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:58.948418 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:58.948666 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 19:26:58.948758 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:26:58.948987 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.936527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.18.229.130:45081)
0924 19:26:58.936710 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.936751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.937782 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.938029 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.938031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.938884 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.938942 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.938943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.939848 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.939848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.941213 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.941215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.941756 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.941898 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.942772 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.942776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.945784 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.947479 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.947481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.947843 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.948058 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.948059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.948059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.948151 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.948450 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.948451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.948494 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.948552 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.948742 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":147,"thread_start_us":95,"threads_started":1}
I20260924 19:26:58.949112 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.936527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:36039)
0924 19:26:58.936933 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.936941 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.937791 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.938029 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.938031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.938540 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.938801 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.938807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.939845 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.939846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.941213 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.941215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.941758 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.941898 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.942551 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.942561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.944895 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.947509 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.947511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.947845 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.948060 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.948060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.948061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.948172 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.948480 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.948481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.948520 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.948911 (+ 391us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.949011 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":51,"thread_start_us":96,"threads_started":1}
I20260924 19:26:58.949450 26564 heartbeater.cc:347] Connected to a master server at 127.18.229.188:43229
I20260924 19:26:58.949548 26564 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.949733 26564 heartbeater.cc:511] Master 127.18.229.188:43229 requested a full tablet report, sending...
I20260924 19:26:58.949846 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:58.936814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.18.229.130:35059)
0924 19:26:58.936968 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:58.936979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:58.937792 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:58.938026 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:58.938030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:58.939065 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:58.939134 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.939137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.940139 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.940140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.941260 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:58.941263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:58.941812 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:58.941813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:58.941900 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:58.942624 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:58.942629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:58.944522 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:58.946606 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.946613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.946615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.946911 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.947134 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:58.947135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:58.947135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:58.947279 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:58.947604 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:58.947611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:58.947773 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 19:26:58.948503 (+ 730us) client_negotiation.cc:240] Negotiation successful
0924 19:26:58.949708 (+ 1205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20260924 19:26:58.949918 26565 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43627
I20260924 19:26:58.949952 26565 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.950016 26565 heartbeater.cc:511] Master 127.18.229.190:43627 requested a full tablet report, sending...
I20260924 19:26:58.950237 26566 heartbeater.cc:347] Connected to a master server at 127.18.229.189:46649
I20260924 19:26:58.950281 26566 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:58.950284 25464 ts_manager.cc:194] Re-registered known tserver with Master: 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447)
I20260924 19:26:58.950341 26566 heartbeater.cc:511] Master 127.18.229.189:46649 requested a full tablet report, sending...
I20260924 19:26:58.950450 26248 ts_manager.cc:194] Registered new tserver with Master: 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447)
I20260924 19:26:58.950767 26062 ts_manager.cc:194] Registered new tserver with Master: 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447)
I20260924 19:26:58.950773 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:36039
W20260924 19:26:59.027629 26571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:26:59.027844 26571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:26:59.027868 26571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:26:59.029043 26571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:26:59.029085 26571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:26:59.029112 26571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:26:59.030577 26571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:39447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=34325
--tserver_master_addrs=127.18.229.190:43627,127.18.229.189:46649,127.18.229.188:43229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:26:59.030825 26571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:26:59.031056 26571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:26:59.036013 26579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:26:59.036010 26577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:59.036058 26571 server_base.cc:1034] running on GCE node
W20260924 19:26:59.036010 26576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:26:59.036409 26571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:26:59.036655 26571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:26:59.037796 26571 hybrid_clock.cc:648] HybridClock initialized: now 1790278019037789 us; error 29 us; skew 500 ppm
Sep 24 19:26:59 dist-test-slave-r8cc krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278019, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:26:59.040228 26571 init.cc:377] Logged in from keytab as oryx/127.18.229.131@KRBTEST.COM (short username oryx)
I20260924 19:26:59.040663 26571 webserver.cc:460] Webserver started at http://127.18.229.131:34325/ using document root <none> and password file <none>
I20260924 19:26:59.040849 26571 fs_manager.cc:359] Metadata directory not provided
I20260924 19:26:59.040899 26571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:26:59.042152 26571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:26:59.042661 26586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:59.042837 26571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:26:59.042924 26571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "2ac4ea199a924848bcc47ca762894dd1"
format_stamp: "Formatted at 2026-09-24 19:26:58 on dist-test-slave-r8cc"
server_key: "8ec7cfe972961249800b6d62989fea03"
server_key_iv: "f081054bac10cbc3bcf9861905cdd7ea"
server_key_version: "encryptionkey@0"
I20260924 19:26:59.043336 26571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:26:59.068372 26571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:26:59.070392 26571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:26:59.070540 26571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:26:59.070791 26571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:26:59.071152 26571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:26:59.071199 26571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:59.071249 26571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:26:59.071274 26571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:26:59.077116 26571 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:39447
I20260924 19:26:59.077168 26699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:39447 every 8 connection(s)
I20260924 19:26:59.077548 26571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:26:59.081970 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 26571
Sep 24 19:26:59 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278019, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.188@KRBTEST.COM
Sep 24 19:26:59 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278019, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
Sep 24 19:26:59 dist-test-slave-r8cc krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278019, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.189@KRBTEST.COM
I20260924 19:26:59.091384 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.078626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:42741 (local address 127.18.229.189:46649)
0924 19:26:59.078675 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.078677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.079768 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 19:26:59.079811 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.079812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.079822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.079872 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.080544 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.081042 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082208 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.082517 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082566 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.088132 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.088133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.088133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.088140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.090014 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090440 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.090441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.090441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.090489 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090678 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.090679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.090679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.090812 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.090896 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.091199 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.091271 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:26:59.091620 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.078257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.131:40537)
0924 19:26:59.078559 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.078568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.079499 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.079828 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.079828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.080216 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:59.080456 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.080462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.081420 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.081421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.082327 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.082329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.082852 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.082921 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.083820 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:59.083825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:59.086189 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:59.090375 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:59.090376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:59.090571 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:59.090778 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:59.090779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:59.090842 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:59.091105 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:59.091105 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:59.091139 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.091199 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.091375 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":55,"thread_start_us":93,"threads_started":1}
I20260924 19:26:59.091763 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.078259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.18.229.131:50885)
0924 19:26:59.078454 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.078498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.079502 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.079752 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.079757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.080209 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:59.080456 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.080462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.081420 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.081421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.082333 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.082335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.082852 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.082921 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.083503 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:59.083509 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:59.086416 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:59.089253 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:59.089260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:59.089262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:59.089569 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:59.089929 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:59.089930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:59.089930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:59.090071 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:59.091482 (+ 1411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:59.091483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:59.091518 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.091575 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.091668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":59,"thread_start_us":116,"threads_started":1}
I20260924 19:26:59.092036 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.078261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:50885 (local address 127.18.229.188:43229)
0924 19:26:59.078337 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.078340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.078587 (+ 247us) server_negotiation.cc:407] Connection header received
0924 19:26:59.079609 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.079610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.079620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.079669 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.080567 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.081323 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082427 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.082730 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082774 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.086502 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.086503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.086503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.086510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.089163 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.089784 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.089785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.089785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.089848 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.091261 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.091262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.091263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.091394 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.091456 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.091857 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.091945 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:26:59.092145 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.078236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:40537 (local address 127.18.229.190:43627)
0924 19:26:59.078289 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.078291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.078617 (+ 326us) server_negotiation.cc:407] Connection header received
0924 19:26:59.079725 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.079726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.079758 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.079810 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.080569 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.081321 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082429 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.082740 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082776 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.086299 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.086300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.086301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.086308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.090293 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090643 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.090644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.090645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.090712 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090908 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.090908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.090909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.091024 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.091080 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.091952 (+ 872us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.092022 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:26:59.092417 26700 heartbeater.cc:347] Connected to a master server at 127.18.229.188:43229
I20260924 19:26:59.092492 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.083208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.18.229.190:43627)
0924 19:26:59.083336 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.083338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.083775 (+ 437us) server_negotiation.cc:407] Connection header received
0924 19:26:59.083801 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.083802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.083816 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.083880 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.084799 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.085542 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.086752 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.087067 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.087164 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.089299 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.089300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.089300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.089307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.091089 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.091474 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.091475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.091475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.091540 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.091728 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.091729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.091729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.091871 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.091965 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.092325 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.092404 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260924 19:26:59.092520 26700 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:59.092604 26701 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43627
I20260924 19:26:59.092643 26701 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:59.092742 26701 heartbeater.cc:511] Master 127.18.229.190:43627 requested a full tablet report, sending...
I20260924 19:26:59.092742 26700 heartbeater.cc:511] Master 127.18.229.188:43229 requested a full tablet report, sending...
I20260924 19:26:59.093113 25464 ts_manager.cc:194] Re-registered known tserver with Master: 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447)
I20260924 19:26:59.093179 26248 ts_manager.cc:194] Registered new tserver with Master: 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447)
I20260924 19:26:59.093600 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:40537
I20260924 19:26:59.093968 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.078564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:46649 (local address 127.18.229.131:42741)
0924 19:26:59.079680 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.079702 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.079760 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.079878 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.079879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.080220 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:26:59.080456 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.080461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.081147 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.081148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.082085 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.082087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.082669 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.082670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.082811 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.083324 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:26:59.083334 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:26:59.088015 (+ 4681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:26:59.090143 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:59.090145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:59.090364 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:59.090556 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:26:59.090557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:26:59.090557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:26:59.090612 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:26:59.090887 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:26:59.090894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:26:59.091063 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.091182 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.093868 (+ 2686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":2568,"thread_start_us":73,"threads_started":1}
I20260924 19:26:59.094261 26702 heartbeater.cc:347] Connected to a master server at 127.18.229.189:46649
I20260924 19:26:59.094300 26702 heartbeater.cc:464] Registering TS with master...
I20260924 19:26:59.094354 26702 heartbeater.cc:511] Master 127.18.229.189:46649 requested a full tablet report, sending...
I20260924 19:26:59.094594 26062 ts_manager.cc:194] Registered new tserver with Master: 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447)
I20260924 19:26:59.094915 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.083521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.18.229.189:46649)
0924 19:26:59.083757 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.083760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.083904 (+ 144us) server_negotiation.cc:407] Connection header received
0924 19:26:59.083924 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.083925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.083935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.083990 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.084798 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.085594 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.086752 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.087054 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.087112 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.088893 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.088894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.088895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.088901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.093509 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.093912 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.093914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.093914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.093972 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.094169 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.094170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.094170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.094284 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.094368 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.094755 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.094832 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":53,"threads_started":1}
I20260924 19:26:59.094081 25464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:26:59.096295 25464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:26:59.101238 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.084449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.18.229.188:43229)
0924 19:26:59.084667 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.084669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.084841 (+ 172us) server_negotiation.cc:407] Connection header received
0924 19:26:59.084862 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.084862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.084874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.084927 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.085700 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.095097 (+ 9397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.095875 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.096229 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.096300 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.097756 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.097757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.097757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.097763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.099540 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.099925 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.099926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.099927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.099988 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.100218 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.100219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.100220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.100357 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.100450 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.101055 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.101144 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":44,"threads_started":1}
I20260924 19:26:59.104242 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.100283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:37889 (local address 127.0.0.1:33886)
0924 19:26:59.100434 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.100445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.100495 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.100985 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.100986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.100989 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:59.101054 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.101055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.102175 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.102176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.103050 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.103052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.103971 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.103972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.104030 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.104034 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.104086 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.104159 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":7}
I20260924 19:26:59.104235 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.100606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.18.229.129:37889)
0924 19:26:59.100799 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.100803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.100815 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:26:59.100859 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.100861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.100880 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.101129 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:59.101204 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.102068 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.103141 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.103878 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.103989 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.104050 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.104101 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.104158 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260924 19:26:59.104951 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.100296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39447 (local address 127.0.0.1:59176)
0924 19:26:59.100609 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.100620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.100656 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.100988 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.100989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.100991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:59.101052 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.101054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.102683 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.102684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.103692 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.103695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.104725 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.104726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.104784 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.104788 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.104853 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.104889 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260924 19:26:59.104959 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.100477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.18.229.131:39447)
0924 19:26:59.100806 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.100810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.100820 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:26:59.100857 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.100860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.100878 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.101286 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:59.101555 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.102596 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.103781 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.104643 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.104746 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.104799 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.104866 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.104904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":88,"threads_started":1}
I20260924 19:26:59.104960 26361 tablet_service.cc:1467] Processing CreateTablet for tablet e3608ccd09534d358cd29f00b4b3d453 (DEFAULT_TABLE table=test-table [id=6917a0c86bf346d6b445b11367650728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:59.105330 26361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3608ccd09534d358cd29f00b4b3d453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:59.105484 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.100409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:37447 (local address 127.0.0.1:48928)
0924 19:26:59.100977 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.100987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.101031 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.101548 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.101549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.101552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:59.101596 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.101598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.103066 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.103067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.103854 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.103856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.105235 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.105236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.105298 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.105302 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.105356 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.105421 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":81,"threads_started":1}
I20260924 19:26:59.105545 26634 tablet_service.cc:1467] Processing CreateTablet for tablet e3608ccd09534d358cd29f00b4b3d453 (DEFAULT_TABLE table=test-table [id=6917a0c86bf346d6b445b11367650728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:59.105762 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.100806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.18.229.130:37447)
0924 19:26:59.101137 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.101141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.101150 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:26:59.101199 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.101202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.101218 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.101286 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:59.101714 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.102975 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.103950 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.104650 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.104745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.104799 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.105624 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.105672 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":86,"threads_started":1}
I20260924 19:26:59.105866 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3608ccd09534d358cd29f00b4b3d453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:59.108037 26726 tablet_bootstrap.cc:492] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Bootstrap starting.
I20260924 19:26:59.108393 26727 tablet_bootstrap.cc:492] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Bootstrap starting.
I20260924 19:26:59.108915 26498 tablet_service.cc:1467] Processing CreateTablet for tablet e3608ccd09534d358cd29f00b4b3d453 (DEFAULT_TABLE table=test-table [id=6917a0c86bf346d6b445b11367650728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:26:59.109017 26727 tablet_bootstrap.cc:654] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:59.109207 26498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3608ccd09534d358cd29f00b4b3d453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:26:59.109332 26727 log.cc:824] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:59.110021 26727 tablet_bootstrap.cc:492] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: No bootstrap required, opened a new log
I20260924 19:26:59.110103 26727 ts_tablet_manager.cc:1397] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:26:59.111989 26729 tablet_bootstrap.cc:492] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Bootstrap starting.
I20260924 19:26:59.111995 26727 raft_consensus.cc:348] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.112200 26727 raft_consensus.cc:374] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:59.112251 26727 raft_consensus.cc:729] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac4ea199a924848bcc47ca762894dd1, State: Initialized, Role: FOLLOWER
I20260924 19:26:59.112268 26726 tablet_bootstrap.cc:654] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:59.112389 26727 consensus_queue.cc:260] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.112599 26726 log.cc:824] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:59.112674 26727 ts_tablet_manager.cc:1428] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:26:59.112803 26701 heartbeater.cc:503] Master 127.18.229.190:43627 was elected leader, sending a full tablet report...
I20260924 19:26:59.112843 26729 tablet_bootstrap.cc:654] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Neither blocks nor log segments found. Creating new log.
I20260924 19:26:59.113133 26729 log.cc:824] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Log is configured to *not* fsync() on all Append() calls
I20260924 19:26:59.113330 26726 tablet_bootstrap.cc:492] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: No bootstrap required, opened a new log
I20260924 19:26:59.113394 26726 ts_tablet_manager.cc:1397] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:26:59.115199 26726 raft_consensus.cc:348] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.115339 26726 raft_consensus.cc:374] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:59.115384 26726 raft_consensus.cc:729] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f885e65c6bdc477a8baa2b67ba10ce08, State: Initialized, Role: FOLLOWER
I20260924 19:26:59.115499 26726 consensus_queue.cc:260] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.115769 26726 ts_tablet_manager.cc:1428] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:59.115778 26428 heartbeater.cc:503] Master 127.18.229.190:43627 was elected leader, sending a full tablet report...
I20260924 19:26:59.116529 26729 tablet_bootstrap.cc:492] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: No bootstrap required, opened a new log
I20260924 19:26:59.116595 26729 ts_tablet_manager.cc:1397] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 19:26:59.117980 26729 raft_consensus.cc:348] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.118122 26729 raft_consensus.cc:374] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:26:59.118166 26729 raft_consensus.cc:729] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fb1d48dca048658a72aa55b3139c85, State: Initialized, Role: FOLLOWER
I20260924 19:26:59.118278 26729 consensus_queue.cc:260] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.118526 26729 ts_tablet_manager.cc:1428] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:26:59.118556 26565 heartbeater.cc:503] Master 127.18.229.190:43627 was elected leader, sending a full tablet report...
W20260924 19:26:59.186564 26567 tablet.cc:2369] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:26:59.245911 26734 raft_consensus.cc:482] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:26:59.246030 26734 raft_consensus.cc:504] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.246487 26734 leader_election.cc:288] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889), 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447)
I20260924 19:26:59.249492 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.246851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:51407 (local address 127.18.229.131:39447)
0924 19:26:59.246899 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.246901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.246909 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:59.246962 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.246962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.246979 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.247042 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:59.247262 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.247745 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.248747 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.249118 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.249223 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.249232 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.249385 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.249425 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:26:59.249492 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.246669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39447 (local address 127.18.229.130:51407)
0924 19:26:59.246832 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.246841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.246902 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.247060 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.247061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.247063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:59.247163 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.247165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.247845 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.247846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.248657 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.248660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.249237 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.249238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.249301 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.249304 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.249377 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.249418 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":50}
I20260924 19:26:59.249555 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.246814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:38699 (local address 127.18.229.129:37889)
0924 19:26:59.246870 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.246872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.246883 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:26:59.246962 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.246962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.246979 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.247042 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:59.247250 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.247729 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.248777 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.249201 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.249287 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.249295 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.249447 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.249480 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:26:59.249600 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.246669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:37889 (local address 127.18.229.130:38699)
0924 19:26:59.246808 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.246821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.246885 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.247060 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.247061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.247063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:59.247163 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.247165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.247817 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.247819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.248672 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.248675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.249287 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.249288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.249361 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.249364 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.249439 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.249480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":40}
I20260924 19:26:59.249903 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3608ccd09534d358cd29f00b4b3d453" candidate_uuid: "52fb1d48dca048658a72aa55b3139c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" is_pre_election: true
I20260924 19:26:59.249903 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3608ccd09534d358cd29f00b4b3d453" candidate_uuid: "52fb1d48dca048658a72aa55b3139c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac4ea199a924848bcc47ca762894dd1" is_pre_election: true
I20260924 19:26:59.250058 26654 raft_consensus.cc:2454] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52fb1d48dca048658a72aa55b3139c85 in term 0.
I20260924 19:26:59.250058 26381 raft_consensus.cc:2454] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52fb1d48dca048658a72aa55b3139c85 in term 0.
I20260924 19:26:59.250283 26453 leader_election.cc:302] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fb1d48dca048658a72aa55b3139c85, f885e65c6bdc477a8baa2b67ba10ce08; no voters:
I20260924 19:26:59.250461 26734 raft_consensus.cc:2781] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:26:59.250499 26734 raft_consensus.cc:482] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:26:59.250514 26734 raft_consensus.cc:3037] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:59.251183 26734 raft_consensus.cc:504] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.251320 26734 leader_election.cc:288] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [CANDIDATE]: Term 1 election: Requested vote from peers f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889), 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447)
I20260924 19:26:59.251487 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3608ccd09534d358cd29f00b4b3d453" candidate_uuid: "52fb1d48dca048658a72aa55b3139c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac4ea199a924848bcc47ca762894dd1"
I20260924 19:26:59.251582 26654 raft_consensus.cc:3037] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:59.251577 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3608ccd09534d358cd29f00b4b3d453" candidate_uuid: "52fb1d48dca048658a72aa55b3139c85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f885e65c6bdc477a8baa2b67ba10ce08"
I20260924 19:26:59.251677 26381 raft_consensus.cc:3037] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:26:59.252497 26654 raft_consensus.cc:2454] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52fb1d48dca048658a72aa55b3139c85 in term 1.
I20260924 19:26:59.253154 26381 raft_consensus.cc:2454] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52fb1d48dca048658a72aa55b3139c85 in term 1.
I20260924 19:26:59.253573 26453 leader_election.cc:302] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fb1d48dca048658a72aa55b3139c85, f885e65c6bdc477a8baa2b67ba10ce08; no voters:
I20260924 19:26:59.253712 26734 raft_consensus.cc:2781] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:26:59.253839 26734 raft_consensus.cc:686] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 1 LEADER]: Becoming Leader. State: Replica: 52fb1d48dca048658a72aa55b3139c85, State: Running, Role: LEADER
I20260924 19:26:59.254048 26734 consensus_queue.cc:237] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } }
I20260924 19:26:59.254873 25464 catalog_manager.cc:5495] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "52fb1d48dca048658a72aa55b3139c85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } health_report { overall_health: HEALTHY } } }
I20260924 19:26:59.260908 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.258625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.18.229.188:43229)
0924 19:26:59.258662 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.258664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.258671 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:26:59.258738 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.258739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.258756 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.258807 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:26:59.258886 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.259349 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.260233 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.260608 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.260669 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.260717 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.260830 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.260858 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:26:59.260929 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.258547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:47634)
0924 19:26:59.258607 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 19:26:59.258620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:26:59.258674 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:26:59.258810 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:26:59.258811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:26:59.258812 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:26:59.258849 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.258851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.259407 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.259407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.260164 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:26:59.260167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.260706 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.260707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:26:59.260760 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.260763 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:26:59.260824 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:26:59.260874 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:26:59.261312 26258 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260924 19:26:59.262153 26737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261791e4f0ef4077b728544ae9a8b4af. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:59.262375 26737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: This master's current role is: LEARNER
I20260924 19:26:59.284365 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.282158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.18.229.130:37447)
0924 19:26:59.282204 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.282206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.282214 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:59.282283 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.282284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.282301 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.282360 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:26:59.282458 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.282976 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.283658 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.283862 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.283910 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.284025 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:26:59.284157 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:26:59.284230 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.284251 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.284268 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:26:59.286314 26654 raft_consensus.cc:1261] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Refusing update from remote peer 52fb1d48dca048658a72aa55b3139c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:59.286307 26381 raft_consensus.cc:1261] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Refusing update from remote peer 52fb1d48dca048658a72aa55b3139c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:26:59.286684 26735 consensus_queue.cc:1035] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:59.286808 26734 consensus_queue.cc:1035] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:26:59.289746 26743 mvcc.cc:205] Tried to move back new op lower bound from 7332978766993915904 to 7332978766865297408. Current Snapshot: MvccSnapshot[applied={T|T < 7332978766993915904}]
I20260924 19:26:59.292380 26744 mvcc.cc:205] Tried to move back new op lower bound from 7332978766993915904 to 7332978766865297408. Current Snapshot: MvccSnapshot[applied={T|T < 7332978766993915904 or (T in {7332978766993915904})}]
I20260924 19:26:59.292461 25464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:59.292562 25464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:26:59.293107 26213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } }
I20260924 19:26:59.293227 26213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:59.294049 26023 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: attempting to promote NON_VOTER ca97866e1f5b4d03978f39b568fb221b to VOTER
I20260924 19:26:59.294165 26023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } }
I20260924 19:26:59.294479 26071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Refusing update from remote peer 261791e4f0ef4077b728544ae9a8b4af: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260924 19:26:59.294732 26258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 LEARNER]: Refusing update from remote peer 261791e4f0ef4077b728544ae9a8b4af: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260924 19:26:59.294787 25464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af: Sending DeleteTablet for 3 replicas of tablet e3608ccd09534d358cd29f00b4b3d453
I20260924 19:26:59.295105 26634 tablet_service.cc:1514] Processing DeleteTablet for tablet e3608ccd09534d358cd29f00b4b3d453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:59 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:59176
I20260924 19:26:59.295195 26361 tablet_service.cc:1514] Processing DeleteTablet for tablet e3608ccd09534d358cd29f00b4b3d453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:59 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:33886
I20260924 19:26:59.295244 26498 tablet_service.cc:1514] Processing DeleteTablet for tablet e3608ccd09534d358cd29f00b4b3d453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:26:59 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:48928
I20260924 19:26:59.295382 26754 tablet_replica.cc:330] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: stopping tablet replica
I20260924 19:26:59.295471 26754 raft_consensus.cc:2229] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:59.295483 26213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260924 19:26:59.295564 26754 raft_consensus.cc:2258] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:59.295600 26213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260924 19:26:59.296029 26755 tablet_replica.cc:330] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: stopping tablet replica
I20260924 19:26:59.296061 26754 ts_tablet_manager.cc:1905] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:59.296190 26755 raft_consensus.cc:2229] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:26:59.296301 26755 raft_consensus.cc:2258] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:59.296612 26753 tablet_replica.cc:330] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: stopping tablet replica
I20260924 19:26:59.296765 26753 raft_consensus.cc:2229] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:26:59.296890 26753 raft_consensus.cc:2258] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:26:59.297319 26753 ts_tablet_manager.cc:1905] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:59.297610 26024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:26:59.297734 26024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:59.297897 26023 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, ca97866e1f5b4d03978f39b568fb221b (127.18.229.188) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:26:59.298092 26754 ts_tablet_manager.cc:1918] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:59.298161 26754 log.cc:1196] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/e3608ccd09534d358cd29f00b4b3d453
I20260924 19:26:59.298450 26754 ts_tablet_manager.cc:1939] T e3608ccd09534d358cd29f00b4b3d453 P f885e65c6bdc477a8baa2b67ba10ce08: Deleting consensus metadata
I20260924 19:26:59.298744 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:26:59.298810 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:59.298866 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "261791e4f0ef4077b728544ae9a8b4af" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:26:59.298913 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261791e4f0ef4077b728544ae9a8b4af [sys.catalog]: This master's current role is: LEADER
I20260924 19:26:59.299009 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25432
I20260924 19:26:59.300508 26755 ts_tablet_manager.cc:1905] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:26:59.302282 26755 ts_tablet_manager.cc:1918] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:59.302358 26755 log.cc:1196] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/e3608ccd09534d358cd29f00b4b3d453
I20260924 19:26:59.302632 26755 ts_tablet_manager.cc:1939] T e3608ccd09534d358cd29f00b4b3d453 P 52fb1d48dca048658a72aa55b3139c85: Deleting consensus metadata
I20260924 19:26:59.305264 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.296978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.18.229.188:43229)
0924 19:26:59.297051 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.297053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.297061 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:26:59.297080 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.297081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.297091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.297141 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.298815 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.299445 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.300256 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.300552 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.300600 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.302386 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.302388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.302388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.302396 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.303957 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.304407 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.304408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.304408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.304462 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.304657 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.304658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.304658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.304776 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.304860 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.305062 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.305146 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:26:59.306106 26753 ts_tablet_manager.cc:1918] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:26:59.306198 26753 log.cc:1196] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/e3608ccd09534d358cd29f00b4b3d453
I20260924 19:26:59.306445 26753 ts_tablet_manager.cc:1939] T e3608ccd09534d358cd29f00b4b3d453 P 2ac4ea199a924848bcc47ca762894dd1: Deleting consensus metadata
W20260924 19:26:59.307005 26452 connection.cc:535] server connection from 127.0.0.1:48928 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:26:59.307018 26588 connection.cc:535] server connection from 127.0.0.1:59176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:26:59.307260 26316 connection.cc:535] server connection from 127.0.0.1:33886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:26:59.316071 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.308751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.18.229.188:43229)
0924 19:26:59.308790 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.308792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.308921 (+ 129us) server_negotiation.cc:407] Connection header received
0924 19:26:59.309002 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.309003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.309013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.309078 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.309814 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.310561 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.311712 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.311956 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.312050 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.313209 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.313211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.313211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.313218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.314842 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.315237 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.315237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.315238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.315283 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.315496 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.315497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.315497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.315605 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.315717 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.315905 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.315969 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:26:59.316642 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:26:59.308896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.18.229.189:46649)
0924 19:26:59.308937 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:26:59.308939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:26:59.309070 (+ 131us) server_negotiation.cc:407] Connection header received
0924 19:26:59.309156 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:26:59.309158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:26:59.309172 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:26:59.309228 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:26:59.309770 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.310560 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.311469 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:26:59.311723 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:26:59.311787 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:26:59.313289 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:26:59.313291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:26:59.313291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:26:59.313298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:26:59.315014 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.315654 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.315655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.315655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.315717 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:26:59.315931 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:26:59.315932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:26:59.315933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:26:59.316065 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:26:59.316181 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 19:26:59.316467 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 19:26:59.316552 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:27:00.291350 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.291059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:35225)
0924 19:27:00.291210 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.291305 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
W20260924 19:27:00.291584 26428 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
I20260924 19:27:00.292579 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.292287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:34623)
0924 19:27:00.292454 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.292540 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260924 19:27:00.292815 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.292606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.131:49829)
0924 19:27:00.292716 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.292775 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
W20260924 19:27:00.292806 26565 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
W20260924 19:27:00.292987 26701 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
I20260924 19:27:00.796535 26767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 261791e4f0ef4077b728544ae9a8b4af)
I20260924 19:27:00.796726 26767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } }
I20260924 19:27:00.797046 26767 leader_election.cc:288] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 261791e4f0ef4077b728544ae9a8b4af (127.18.229.190:43627), ca97866e1f5b4d03978f39b568fb221b (127.18.229.188:43229)
I20260924 19:27:00.797429 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.797214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:44800)
0924 19:27:00.797330 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.797384 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":62,"threads_started":1}
W20260924 19:27:00.797608 26048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
I20260924 19:27:00.798029 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.797911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:44810)
0924 19:27:00.797979 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.798005 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:27:00.798095 26048 leader_election.cc:334] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 261791e4f0ef4077b728544ae9a8b4af (127.18.229.190:43627): Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
I20260924 19:27:00.800271 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.797221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43229 (local address 127.0.0.1:50428)
0924 19:27:00.797350 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.797368 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:00.797415 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:00.797722 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:00.797723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:00.797725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:00.797787 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.797790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.798616 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.798617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.799487 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.799490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.800035 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.800036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.800092 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.800095 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:00.800186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:27:00.800226 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":38,"threads_started":1}
I20260924 19:27:00.800277 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.797378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.18.229.188:43229)
0924 19:27:00.797504 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.797539 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.797548 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:00.797577 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.797578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.797589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.797652 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:00.797873 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.798532 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.799590 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.799968 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.800028 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.800035 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.800189 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.800225 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260924 19:27:00.800551 26258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca97866e1f5b4d03978f39b568fb221b" is_pre_election: true
I20260924 19:27:00.800673 26258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80f4a776d1e14bbbb3d1b06f9155258b in term 1.
I20260924 19:27:00.800848 26046 leader_election.cc:302] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f4a776d1e14bbbb3d1b06f9155258b, ca97866e1f5b4d03978f39b568fb221b; no voters: 261791e4f0ef4077b728544ae9a8b4af
I20260924 19:27:00.800957 26767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:27:00.801005 26767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 261791e4f0ef4077b728544ae9a8b4af)
I20260924 19:27:00.801034 26767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:27:00.801725 26767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } }
I20260924 19:27:00.801875 26767 leader_election.cc:288] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [CANDIDATE]: Term 2 election: Requested vote from peers 261791e4f0ef4077b728544ae9a8b4af (127.18.229.190:43627), ca97866e1f5b4d03978f39b568fb221b (127.18.229.188:43229)
I20260924 19:27:00.802094 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.801942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:44814)
0924 19:27:00.802040 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.802073 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:27:00.802053 26258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca97866e1f5b4d03978f39b568fb221b"
I20260924 19:27:00.802134 26258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:27:00.802433 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.802305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:44820)
0924 19:27:00.802383 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.802412 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 19:27:00.802515 26048 leader_election.cc:334] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 261791e4f0ef4077b728544ae9a8b4af (127.18.229.190:43627): Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
I20260924 19:27:00.802863 26258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f4a776d1e14bbbb3d1b06f9155258b in term 2.
I20260924 19:27:00.803035 26046 leader_election.cc:302] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f4a776d1e14bbbb3d1b06f9155258b, ca97866e1f5b4d03978f39b568fb221b; no voters: 261791e4f0ef4077b728544ae9a8b4af
I20260924 19:27:00.803117 26767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:27:00.803252 26767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 2 LEADER]: Becoming Leader. State: Replica: 80f4a776d1e14bbbb3d1b06f9155258b, State: Running, Role: LEADER
I20260924 19:27:00.803335 26767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } }
I20260924 19:27:00.803709 26772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f4a776d1e14bbbb3d1b06f9155258b. Latest consensus state: current_term: 2 leader_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.803812 26772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:00.803961 26774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:00.804356 26774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:00.804412 26774 catalog_manager.cc:1259] Loaded cluster ID: 9e3e6509a910435389bb4c6ff093c579
I20260924 19:27:00.804428 26774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:27:00.804550 26774 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:00.804607 26774 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Loaded TSK: 0
I20260924 19:27:00.804694 26774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:27:00.860205 26062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:27:00.861852 26062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:00.862902 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.862779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:44842)
0924 19:27:00.862838 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.862873 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:27:00.862892 26258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 2 FOLLOWER]: Refusing update from remote peer 80f4a776d1e14bbbb3d1b06f9155258b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 19:27:00.863222 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.863127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.0.0.1:44856)
0924 19:27:00.863161 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.863204 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 19:27:00.863164 26767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260924 19:27:00.863305 26048 consensus_peers.cc:487] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b -> Peer 261791e4f0ef4077b728544ae9a8b4af (127.18.229.190:43627): Couldn't send request to peer 261791e4f0ef4077b728544ae9a8b4af. Status: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:27:00.863794 26776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f4a776d1e14bbbb3d1b06f9155258b. Latest consensus state: current_term: 2 leader_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: NON_VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.863916 26776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:27:00.864185 26772 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [term 2 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, ca97866e1f5b4d03978f39b568fb221b (127.18.229.188) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.864274 26258 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [term 2 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, ca97866e1f5b4d03978f39b568fb221b (127.18.229.188) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.865214 26772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.865300 26772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:00.865391 26767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.865448 26767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:00.865260 26780 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.865505 26776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "261791e4f0ef4077b728544ae9a8b4af" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43627 } } peers { permanent_uuid: "80f4a776d1e14bbbb3d1b06f9155258b" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 46649 } attrs { promote: false } } peers { permanent_uuid: "ca97866e1f5b4d03978f39b568fb221b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43229 } attrs { promote: false } } }
I20260924 19:27:00.865543 26780 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:27:00.865571 26776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca97866e1f5b4d03978f39b568fb221b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:27:00.865298 26775 mvcc.cc:205] Tried to move back new op lower bound from 7332978773452349440 to 7332978773211156480. Current Snapshot: MvccSnapshot[applied={T|T < 7332978773452349440 or (T in {7332978773452349440})}]
I20260924 19:27:00.870601 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.867579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.18.229.129:37889)
0924 19:27:00.867785 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.867787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.867795 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:00.867821 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.867822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.867831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.868013 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:00.868048 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.868771 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.869750 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.870310 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.870386 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.870396 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.870484 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.870525 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
I20260924 19:27:00.870617 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.867486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:37889 (local address 127.0.0.1:51656)
0924 19:27:00.867600 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.867610 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:00.867652 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:00.867918 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:00.867918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:00.867920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:00.867955 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.867956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.868857 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.868858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.869651 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.869653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.870387 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.870388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.870432 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.870436 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:27:00.870479 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:27:00.870539 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":79,"threads_started":1}
I20260924 19:27:00.870754 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.867265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39447 (local address 127.0.0.1:44814)
0924 19:27:00.867293 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.867304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:00.867345 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:00.867652 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:00.867653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:00.867655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:00.867708 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.867710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.868721 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.868721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.869804 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.869806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.870529 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.870530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.870582 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.870585 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:00.870650 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:27:00.870698 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":8}
I20260924 19:27:00.871096 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.867342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.18.229.131:39447)
0924 19:27:00.867506 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.867508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.867518 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:27:00.867544 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.867545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.867556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.867786 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:00.867829 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.868645 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.869887 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.870458 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.870525 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.870532 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.870937 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.871022 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":52,"threads_started":1}
I20260924 19:27:00.871172 26361 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc3f48811c64315a8b5816f00e2561a (DEFAULT_TABLE table=test-table [id=1c7251fe1f1a4fc7a2f5b5266ea9fd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:00.871317 26361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc3f48811c64315a8b5816f00e2561a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:00.871313 26634 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc3f48811c64315a8b5816f00e2561a (DEFAULT_TABLE table=test-table [id=1c7251fe1f1a4fc7a2f5b5266ea9fd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:00.871421 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc3f48811c64315a8b5816f00e2561a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:00.871615 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.867357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:37447 (local address 127.0.0.1:59816)
0924 19:27:00.867392 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.867401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:00.867438 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:00.867901 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:00.867902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:00.867904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:00.867964 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.867966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.868638 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.868639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.869806 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.869807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.871374 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.871375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.871446 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.871450 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:27:00.871512 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:27:00.871561 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":6}
I20260924 19:27:00.871628 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.867656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.18.229.130:37447)
0924 19:27:00.867772 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.867774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.867782 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:00.867805 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.867806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.867815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.867900 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:00.868030 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.868560 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.870564 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.871304 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.871377 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.871386 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.871528 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.871568 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260924 19:27:00.872749 26787 tablet_bootstrap.cc:492] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08: Bootstrap starting.
I20260924 19:27:00.872901 26788 tablet_bootstrap.cc:492] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Bootstrap starting.
I20260924 19:27:00.872970 26498 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc3f48811c64315a8b5816f00e2561a (DEFAULT_TABLE table=test-table [id=1c7251fe1f1a4fc7a2f5b5266ea9fd6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:00.873088 26498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc3f48811c64315a8b5816f00e2561a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:00.873274 26788 tablet_bootstrap.cc:654] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:00.873453 26787 tablet_bootstrap.cc:654] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:00.873835 26788 tablet_bootstrap.cc:492] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: No bootstrap required, opened a new log
I20260924 19:27:00.873884 26788 ts_tablet_manager.cc:1397] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:00.874065 26787 tablet_bootstrap.cc:492] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08: No bootstrap required, opened a new log
I20260924 19:27:00.874047 26788 raft_consensus.cc:348] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.874119 26788 raft_consensus.cc:374] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:00.874133 26787 ts_tablet_manager.cc:1397] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:00.874153 26788 raft_consensus.cc:729] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac4ea199a924848bcc47ca762894dd1, State: Initialized, Role: FOLLOWER
I20260924 19:27:00.874210 26788 consensus_queue.cc:260] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.874285 26787 raft_consensus.cc:348] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.874356 26787 raft_consensus.cc:374] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:00.874377 26787 raft_consensus.cc:729] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f885e65c6bdc477a8baa2b67ba10ce08, State: Initialized, Role: FOLLOWER
I20260924 19:27:00.874377 26788 ts_tablet_manager.cc:1428] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:00.874435 26787 consensus_queue.cc:260] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.874591 26787 ts_tablet_manager.cc:1428] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:00.874888 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.874575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.131:53469)
0924 19:27:00.874801 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.874840 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":71,"threads_started":1}
I20260924 19:27:00.875156 26429 heartbeater.cc:503] Master 127.18.229.189:46649 was elected leader, sending a full tablet report...
I20260924 19:27:00.875191 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.874769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:33291)
0924 19:27:00.875102 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.875139 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":65,"threads_started":1}
I20260924 19:27:00.875455 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.875381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:38265)
0924 19:27:00.875413 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.875435 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 19:27:00.875566 26791 tablet_bootstrap.cc:492] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Bootstrap starting.
I20260924 19:27:00.875909 26791 tablet_bootstrap.cc:654] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:00.876454 26791 tablet_bootstrap.cc:492] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: No bootstrap required, opened a new log
I20260924 19:27:00.876495 26791 ts_tablet_manager.cc:1397] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:00.876569 26791 raft_consensus.cc:348] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.876616 26791 raft_consensus.cc:374] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:00.876629 26791 raft_consensus.cc:729] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fb1d48dca048658a72aa55b3139c85, State: Initialized, Role: FOLLOWER
I20260924 19:27:00.876694 26791 consensus_queue.cc:260] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.876852 26791 ts_tablet_manager.cc:1428] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:00.877210 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.877006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:34487)
0924 19:27:00.877137 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.877173 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":92,"threads_started":1}
I20260924 19:27:00.877411 26566 heartbeater.cc:503] Master 127.18.229.189:46649 was elected leader, sending a full tablet report...
I20260924 19:27:00.877542 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.877461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:51947)
0924 19:27:00.877493 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.877522 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 19:27:00.933858 26793 raft_consensus.cc:482] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:00.933959 26793 raft_consensus.cc:504] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.934346 26793 leader_election.cc:288] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447), 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447)
I20260924 19:27:00.937333 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.934618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:46939 (local address 127.18.229.131:39447)
0924 19:27:00.934665 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.934667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.934680 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:27:00.934753 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.934754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.934764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.934832 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:00.935034 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.935535 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.936627 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.936980 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.937070 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.937080 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.937233 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.937270 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:27:00.937338 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.934537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39447 (local address 127.18.229.129:46939)
0924 19:27:00.934607 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.934620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:00.934687 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:00.934852 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:00.934853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:00.934856 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:00.934949 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.934951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.935628 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.935629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.936531 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.936533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.937068 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.937069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.937144 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.937147 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:00.937224 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:27:00.937263 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":27}
I20260924 19:27:00.937616 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.934537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:37447 (local address 127.18.229.129:56903)
0924 19:27:00.934741 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.934752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:00.934825 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:00.934943 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:00.934943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:00.934945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:00.935018 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.935020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.935699 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.935700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.936824 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:00.936826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.937377 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.937378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:00.937451 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.937456 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:27:00.937510 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:27:00.937543 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":41,"thread_start_us":109,"threads_started":1}
I20260924 19:27:00.937654 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.934620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:56903 (local address 127.18.229.130:37447)
0924 19:27:00.934665 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.934667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.934839 (+ 172us) server_negotiation.cc:407] Connection header received
0924 19:27:00.934872 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.934873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.934883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.934934 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:00.935089 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.935589 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.936923 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.937294 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.937377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.937386 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.937536 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.937588 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:27:00.937736 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc3f48811c64315a8b5816f00e2561a" candidate_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac4ea199a924848bcc47ca762894dd1" is_pre_election: true
I20260924 19:27:00.937821 26654 raft_consensus.cc:2454] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f885e65c6bdc477a8baa2b67ba10ce08 in term 0.
I20260924 19:27:00.937899 26518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc3f48811c64315a8b5816f00e2561a" candidate_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fb1d48dca048658a72aa55b3139c85" is_pre_election: true
I20260924 19:27:00.938021 26518 raft_consensus.cc:2454] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f885e65c6bdc477a8baa2b67ba10ce08 in term 0.
I20260924 19:27:00.938055 26315 leader_election.cc:302] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac4ea199a924848bcc47ca762894dd1, f885e65c6bdc477a8baa2b67ba10ce08; no voters:
I20260924 19:27:00.938213 26793 raft_consensus.cc:2781] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:00.938252 26793 raft_consensus.cc:482] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:00.938267 26793 raft_consensus.cc:3037] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:00.938927 26793 raft_consensus.cc:504] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.939072 26793 leader_election.cc:288] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [CANDIDATE]: Term 1 election: Requested vote from peers 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447), 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447)
I20260924 19:27:00.939221 26654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc3f48811c64315a8b5816f00e2561a" candidate_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac4ea199a924848bcc47ca762894dd1"
I20260924 19:27:00.939229 26518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc3f48811c64315a8b5816f00e2561a" candidate_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fb1d48dca048658a72aa55b3139c85"
I20260924 19:27:00.939296 26518 raft_consensus.cc:3037] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:00.939316 26654 raft_consensus.cc:3037] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:00.939932 26518 raft_consensus.cc:2454] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f885e65c6bdc477a8baa2b67ba10ce08 in term 1.
I20260924 19:27:00.939936 26654 raft_consensus.cc:2454] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f885e65c6bdc477a8baa2b67ba10ce08 in term 1.
I20260924 19:27:00.940125 26316 leader_election.cc:302] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fb1d48dca048658a72aa55b3139c85, f885e65c6bdc477a8baa2b67ba10ce08; no voters:
I20260924 19:27:00.940231 26793 raft_consensus.cc:2781] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:00.940382 26793 raft_consensus.cc:686] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 LEADER]: Becoming Leader. State: Replica: f885e65c6bdc477a8baa2b67ba10ce08, State: Running, Role: LEADER
I20260924 19:27:00.940454 26793 consensus_queue.cc:237] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } }
I20260924 19:27:00.941069 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.940927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:46007)
0924 19:27:00.941008 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.941036 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:27:00.941329 26061 catalog_manager.cc:5495] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 reported cstate change: term changed from 0 to 1, leader changed from <none> to f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129). New cstate: current_term: 1 leader_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f885e65c6bdc477a8baa2b67ba10ce08" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37889 } health_report { overall_health: HEALTHY } } }
I20260924 19:27:00.963212 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.960879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.18.229.129:37889)
0924 19:27:00.960927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:00.960930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:00.960941 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:27:00.960991 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:00.960992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:00.961010 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:00.961089 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:00.961219 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.961688 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.962457 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:00.962649 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:00.962731 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:00.962875 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:00.962971 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:00.963046 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:00.963100 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:27:00.963120 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:27:00.964272 26654 raft_consensus.cc:1261] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Refusing update from remote peer f885e65c6bdc477a8baa2b67ba10ce08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:00.964545 26518 raft_consensus.cc:1261] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Refusing update from remote peer f885e65c6bdc477a8baa2b67ba10ce08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:00.964550 26800 consensus_queue.cc:1035] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac4ea199a924848bcc47ca762894dd1" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:00.964828 26793 consensus_queue.cc:1035] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fb1d48dca048658a72aa55b3139c85" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:00.965062 26702 heartbeater.cc:503] Master 127.18.229.189:46649 was elected leader, sending a full tablet report...
I20260924 19:27:00.965440 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.965094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.131:55017)
0924 19:27:00.965366 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.965408 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20260924 19:27:00.965834 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.965709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:35831)
0924 19:27:00.965774 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.965802 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 19:27:00.966507 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.966397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:49535)
0924 19:27:00.966447 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.966479 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:27:00.967176 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.967046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.131:60405)
0924 19:27:00.967127 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.967151 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:27:00.967767 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.967358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.129:54991)
0924 19:27:00.967714 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.967744 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
I20260924 19:27:00.967937 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.967818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43627 (local address 127.18.229.130:60791)
0924 19:27:00.967891 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.967913 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 19:27:00.968784 26062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:00.968874 26062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:00.970276 26062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80f4a776d1e14bbbb3d1b06f9155258b: Sending DeleteTablet for 3 replicas of tablet 4cc3f48811c64315a8b5816f00e2561a
I20260924 19:27:00.970538 26634 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc3f48811c64315a8b5816f00e2561a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:00 UTC) from {username='oryx', principal='oryx/127.18.229.189@KRBTEST.COM'} at 127.0.0.1:44814
I20260924 19:27:00.970577 26361 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc3f48811c64315a8b5816f00e2561a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:00 UTC) from {username='oryx', principal='oryx/127.18.229.189@KRBTEST.COM'} at 127.0.0.1:51656
I20260924 19:27:00.970773 26813 tablet_replica.cc:330] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08: stopping tablet replica
I20260924 19:27:00.970847 26813 raft_consensus.cc:2229] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:27:00.970860 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 26209
I20260924 19:27:00.970961 26813 raft_consensus.cc:2258] T 4cc3f48811c64315a8b5816f00e2561a P f885e65c6bdc477a8baa2b67ba10ce08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:00.970947 26498 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc3f48811c64315a8b5816f00e2561a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:00 UTC) from {username='oryx', principal='oryx/127.18.229.189@KRBTEST.COM'} at 127.0.0.1:59816
I20260924 19:27:00.971107 26812 tablet_replica.cc:330] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: stopping tablet replica
I20260924 19:27:00.971155 26812 raft_consensus.cc:2229] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:00.971202 26812 raft_consensus.cc:2258] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:00.971339 26814 tablet_replica.cc:330] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: stopping tablet replica
I20260924 19:27:00.971422 26814 raft_consensus.cc:2229] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:00.971467 26814 raft_consensus.cc:2258] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:00.971709 26814 ts_tablet_manager.cc:1905] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:00.973286 26812 ts_tablet_manager.cc:1905] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:00.974436 26812 ts_tablet_manager.cc:1918] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:00.974495 26812 log.cc:1196] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/4cc3f48811c64315a8b5816f00e2561a
I20260924 19:27:00.974730 26812 ts_tablet_manager.cc:1939] T 4cc3f48811c64315a8b5816f00e2561a P 2ac4ea199a924848bcc47ca762894dd1: Deleting consensus metadata
I20260924 19:27:00.974722 26814 ts_tablet_manager.cc:1918] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:00.974779 26814 log.cc:1196] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/4cc3f48811c64315a8b5816f00e2561a
I20260924 19:27:00.975021 26814 ts_tablet_manager.cc:1939] T 4cc3f48811c64315a8b5816f00e2561a P 52fb1d48dca048658a72aa55b3139c85: Deleting consensus metadata
I20260924 19:27:00.975040 26047 catalog_manager.cc:4841] TS 2ac4ea199a924848bcc47ca762894dd1 (127.18.229.131:39447): tablet 4cc3f48811c64315a8b5816f00e2561a (table test-table [id=1c7251fe1f1a4fc7a2f5b5266ea9fd6f]) successfully deleted
I20260924 19:27:00.975317 26048 catalog_manager.cc:4841] TS 52fb1d48dca048658a72aa55b3139c85 (127.18.229.130:37447): tablet 4cc3f48811c64315a8b5816f00e2561a (table test-table [id=1c7251fe1f1a4fc7a2f5b5266ea9fd6f]) successfully deleted
I20260924 19:27:00.976989 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:00.976853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:37889 (local address 127.0.0.1:51678)
0924 19:27:00.976922 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:00.976951 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 19:27:00.977072 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 26435
W20260924 19:27:00.977130 26048 catalog_manager.cc:4584] TS f885e65c6bdc477a8baa2b67ba10ce08 (127.18.229.129:37889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cc3f48811c64315a8b5816f00e2561a: Network error: Client connection negotiation failed: client connection to 127.18.229.129:37889: connect: Connection refused (error 111)
I20260924 19:27:00.983387 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 26571
I20260924 19:27:00.988445 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 25928
I20260924 19:27:00.993161 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 26116
2026-09-24T19:27:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4663 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:27:01 dist-test-slave-r8cc krb5kdc[26817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:27:01 dist-test-slave-r8cc krb5kdc[26817](info): set up 2 sockets
Sep 24 19:27:01 dist-test-slave-r8cc krb5kdc[26817](info): commencing operation
krb5kdc: starting...
W20260924 19:27:02.028415 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab.
Entry for principal oryx/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab.
I20260924 19:27:02.139077 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:40393
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab
--principal=oryx/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:02.219851 26830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:02.220037 26830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:02.220080 26830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:02.221253 26830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:02.221300 26830 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:27:02.221313 26830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:02.221326 26830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:02.221338 26830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:02.221350 26830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:02.222868 26830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:40393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.190.keytab
--principal=oryx/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:02.223112 26830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:02.223336 26830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:02.228133 26835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:02.228221 26836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:02.228113 26838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.228504 26830 server_base.cc:1034] running on GCE node
I20260924 19:27:02.228714 26830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:02.228987 26830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:02.230132 26830 hybrid_clock.cc:648] HybridClock initialized: now 1790278022230123 us; error 29 us; skew 500 ppm
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:02.232765 26830 init.cc:377] Logged in from keytab as oryx/127.18.229.190@KRBTEST.COM (short username oryx)
I20260924 19:27:02.233196 26830 webserver.cc:460] Webserver started at http://127.18.229.190:37323/ using document root <none> and password file <none>
I20260924 19:27:02.233438 26830 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:02.233482 26830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:02.233621 26830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:02.234655 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "322a95f6e118442e98baa51f12b5d7ba"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "c5a52d1ae1b291cf374e6abd9502a2e8"
server_key_iv: "cf24c1fa1653a0d766a1ac6a5d828ab9"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.235009 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "322a95f6e118442e98baa51f12b5d7ba"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "c5a52d1ae1b291cf374e6abd9502a2e8"
server_key_iv: "cf24c1fa1653a0d766a1ac6a5d828ab9"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.236311 26830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:27:02.236974 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.237133 26830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:02.237247 26830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "322a95f6e118442e98baa51f12b5d7ba"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "c5a52d1ae1b291cf374e6abd9502a2e8"
server_key_iv: "cf24c1fa1653a0d766a1ac6a5d828ab9"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.237320 26830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:02.276964 26830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:02.279395 26830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:02.279564 26830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:02.284009 26830 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:40393
I20260924 19:27:02.284044 26907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:40393 every 8 connection(s)
I20260924 19:27:02.284415 26830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:27:02.285063 26908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:02.287187 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Bootstrap starting.
I20260924 19:27:02.287796 26908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:02.288053 26908 log.cc:824] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:02.288699 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: No bootstrap required, opened a new log
I20260924 19:27:02.290628 26908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322a95f6e118442e98baa51f12b5d7ba" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40393 } }
I20260924 19:27:02.290763 26908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:02.290786 26908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322a95f6e118442e98baa51f12b5d7ba, State: Initialized, Role: FOLLOWER
I20260924 19:27:02.290916 26908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322a95f6e118442e98baa51f12b5d7ba" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40393 } }
I20260924 19:27:02.290999 26908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:27:02.291023 26908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:27:02.291049 26908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:02.291507 26908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322a95f6e118442e98baa51f12b5d7ba" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40393 } }
I20260924 19:27:02.291585 26908 leader_election.cc:302] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322a95f6e118442e98baa51f12b5d7ba; no voters:
I20260924 19:27:02.291702 26908 leader_election.cc:288] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:27:02.291774 26911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:02.291934 26908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:27:02.291946 26911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [term 1 LEADER]: Becoming Leader. State: Replica: 322a95f6e118442e98baa51f12b5d7ba, State: Running, Role: LEADER
I20260924 19:27:02.292169 26911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322a95f6e118442e98baa51f12b5d7ba" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40393 } }
I20260924 19:27:02.292541 26912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "322a95f6e118442e98baa51f12b5d7ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322a95f6e118442e98baa51f12b5d7ba" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40393 } } }
I20260924 19:27:02.292551 26913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322a95f6e118442e98baa51f12b5d7ba. Latest consensus state: current_term: 1 leader_uuid: "322a95f6e118442e98baa51f12b5d7ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322a95f6e118442e98baa51f12b5d7ba" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40393 } } }
I20260924 19:27:02.292644 26912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:02.292654 26913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:02.293008 26920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:02.293540 26920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:02.294082 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 26830
I20260924 19:27:02.294199 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:27:02.294395 19350 external_mini_cluster.cc:1428] Setting key ef8f0730cb98bbe51d644097bf2888c2
I20260924 19:27:02.295086 26920 catalog_manager.cc:1347] Generated new cluster ID: 9869266269f34048bb5ec0d6889e82cd
I20260924 19:27:02.295138 26920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:02.304965 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.295005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.18.229.190:40393)
0924 19:27:02.296292 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.296299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.296322 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:27:02.296409 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.296414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.296454 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.296588 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.297195 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.298091 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.298854 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.299264 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.299392 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.301747 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.301750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.301751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.301769 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.303348 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.303723 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.303726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.303726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.303813 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.304019 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.304020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.304020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.304317 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.304483 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.304564 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.304797 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1249,"thread_start_us":81,"threads_started":1}
I20260924 19:27:02.320538 26920 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:27:02.321080 26920 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:02.327580 26920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Generated new TSK 0
I20260924 19:27:02.327780 26920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab.
Entry for principal oryx/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab.
I20260924 19:27:02.355171 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40393
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:02.446053 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:02.446241 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:02.446264 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:02.447402 26934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:02.447441 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:02.447486 26934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:27:02.447522 26934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:27:02.449059 26934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:40393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:02.449306 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:02.449530 26934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:02.454475 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:02.454459 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.454476 26934 server_base.cc:1034] running on GCE node
W20260924 19:27:02.454450 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.454885 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:02.455111 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:02.456254 26934 hybrid_clock.cc:648] HybridClock initialized: now 1790278022456236 us; error 29 us; skew 500 ppm
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:02.458591 26934 init.cc:377] Logged in from keytab as oryx/127.18.229.129@KRBTEST.COM (short username oryx)
I20260924 19:27:02.459002 26934 webserver.cc:460] Webserver started at http://127.18.229.129:36969/ using document root <none> and password file <none>
I20260924 19:27:02.459230 26934 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:02.459297 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:02.459448 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:02.460554 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "bfad6ba329eb41659c4e2325d94919fa"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "d43e47e6722936dbaf1821beefe5c2dd"
server_key_iv: "5647d8cba287aeda143f1d0be1668649"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.460917 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "bfad6ba329eb41659c4e2325d94919fa"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "d43e47e6722936dbaf1821beefe5c2dd"
server_key_iv: "5647d8cba287aeda143f1d0be1668649"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.462206 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:27:02.462870 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.463032 26934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:02.463125 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "bfad6ba329eb41659c4e2325d94919fa"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "d43e47e6722936dbaf1821beefe5c2dd"
server_key_iv: "5647d8cba287aeda143f1d0be1668649"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.463205 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:02.471865 26934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:02.473574 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:02.473729 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:02.474189 26934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:02.474258 26934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.474298 26934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:02.474324 26934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:02.480911 26934 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:40289
I20260924 19:27:02.481242 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:27:02.481395 27064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:40289 every 8 connection(s)
I20260924 19:27:02.483949 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.474624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:52945 (local address 127.18.229.190:40393)
0924 19:27:02.474697 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.474699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.474906 (+ 207us) server_negotiation.cc:407] Connection header received
0924 19:27:02.475336 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.475338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.475352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.475437 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.476286 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.476793 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.478367 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.478757 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.478843 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.481647 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.481649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.481649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.481661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.482855 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.483203 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.483204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.483204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.483253 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.483450 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.483451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.483451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.483547 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.483596 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.483820 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.483888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:27:02.484079 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.474601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.18.229.129:52945)
0924 19:27:02.474770 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.474807 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.475247 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.475519 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.475523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.475885 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:02.476187 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.476194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.476889 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.476890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.478267 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.478271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.478865 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.478866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.479003 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.479538 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:02.479551 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:02.481537 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:02.482924 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.482930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.482931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.483140 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.483308 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.483308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.483308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.483399 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.483598 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:02.483602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:02.483715 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.483793 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.483904 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260924 19:27:02.484581 27065 heartbeater.cc:347] Connected to a master server at 127.18.229.190:40393
I20260924 19:27:02.484686 27065 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:02.484853 27065 heartbeater.cc:511] Master 127.18.229.190:40393 requested a full tablet report, sending...
I20260924 19:27:02.485412 26861 ts_manager.cc:194] Registered new tserver with Master: bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289)
I20260924 19:27:02.486338 26861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:52945
I20260924 19:27:02.490038 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 26934
I20260924 19:27:02.490152 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:27:02.490386 19350 external_mini_cluster.cc:1428] Setting key fe146dcc58031cf185320b94c5cfe8f7
I20260924 19:27:02.490658 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.485543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.18.229.129:45835)
0924 19:27:02.485709 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.485722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.485783 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.485903 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.485904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.486083 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:02.486134 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.486135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.486880 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.486881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.487389 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.487390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.487696 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.487697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.487744 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.488188 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:02.488193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:02.488459 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:02.489643 (+ 1184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.489643 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.489644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.489797 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.489982 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.489983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.489983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.490048 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.490326 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:02.490327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:02.490376 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.490450 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.490553 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":17,"thread_start_us":80,"threads_started":1}
I20260924 19:27:02.490666 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.485579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:45835 (local address 127.18.229.190:40393)
0924 19:27:02.485623 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.485625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.485783 (+ 158us) server_negotiation.cc:407] Connection header received
0924 19:27:02.485838 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.485839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.485847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.485888 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.486197 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.486790 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.487458 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.487631 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.487687 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.488523 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.488524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.488525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.488529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.489576 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.489873 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.489874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.489874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.489917 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.490130 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.490131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.490131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.490258 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.490309 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.490475 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.490564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab.
Entry for principal oryx/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab.
I20260924 19:27:02.512951 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40393
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:02.593228 27074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:02.593416 27074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:02.593439 27074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:02.594579 27074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:02.594616 27074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:02.594636 27074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:27:02.594653 27074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:27:02.596158 27074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:40393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:02.596331 27074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:02.596488 27074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:02.601261 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.601426 27074 server_base.cc:1034] running on GCE node
W20260924 19:27:02.601240 27080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:02.601239 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.601673 27074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:02.601905 27074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:02.603034 27074 hybrid_clock.cc:648] HybridClock initialized: now 1790278022603025 us; error 26 us; skew 500 ppm
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:02.605427 27074 init.cc:377] Logged in from keytab as oryx/127.18.229.130@KRBTEST.COM (short username oryx)
I20260924 19:27:02.605839 27074 webserver.cc:460] Webserver started at http://127.18.229.130:42079/ using document root <none> and password file <none>
I20260924 19:27:02.606070 27074 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:02.606138 27074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:02.606288 27074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:02.607306 27074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "af924e7852cbfb0d0622b8cd0c4de22a"
server_key_iv: "a7dcf4abfc18fda5274ba257fb653b60"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.607638 27074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "af924e7852cbfb0d0622b8cd0c4de22a"
server_key_iv: "a7dcf4abfc18fda5274ba257fb653b60"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.609083 27074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:27:02.609791 27089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.609992 27074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.610090 27074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "af924e7852cbfb0d0622b8cd0c4de22a"
server_key_iv: "a7dcf4abfc18fda5274ba257fb653b60"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.610174 27074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:02.620589 27074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:02.622300 27074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:02.622455 27074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:02.622911 27074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:02.622972 27074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.623018 27074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:02.623040 27074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.629565 27074 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41455
I20260924 19:27:02.629631 27204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41455 every 8 connection(s)
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:02.629915 27074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:27:02.632918 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.623422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:36725 (local address 127.18.229.190:40393)
0924 19:27:02.623461 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.623463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.623690 (+ 227us) server_negotiation.cc:407] Connection header received
0924 19:27:02.624236 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.624237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.624252 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.624301 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.625069 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.625886 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.627442 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.627783 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.627889 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.630440 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.630441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.630442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.630448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.631854 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.632244 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.632244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.632245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.632289 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.632447 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.632448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.632448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.632549 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.632593 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.632802 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.632857 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:27:02.633144 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.623410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.18.229.130:36725)
0924 19:27:02.623574 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.623606 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.624089 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.624367 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.624370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.624691 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:02.624974 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.624980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.625963 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.625964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.627337 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.627340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.627891 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.627892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.627993 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.628596 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:02.628606 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:02.630361 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:02.631921 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.631926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.631927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.632179 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.632328 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.632328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.632329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.632411 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.632599 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:02.632603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:02.632710 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.632785 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.632895 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260924 19:27:02.633641 27205 heartbeater.cc:347] Connected to a master server at 127.18.229.190:40393
I20260924 19:27:02.633736 27205 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:02.633916 27205 heartbeater.cc:511] Master 127.18.229.190:40393 requested a full tablet report, sending...
I20260924 19:27:02.634590 26862 ts_manager.cc:194] Registered new tserver with Master: 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455)
I20260924 19:27:02.634974 26862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:36725
I20260924 19:27:02.637748 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 27074
I20260924 19:27:02.637874 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:27:02.638119 19350 external_mini_cluster.cc:1428] Setting key 85b8645278e1d1272c0892e72667c800
I20260924 19:27:02.640410 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.634624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.18.229.130:46277)
0924 19:27:02.635133 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.635142 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.635199 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.635318 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.635318 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.635505 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:02.635565 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.635567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.636206 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.636206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.636677 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.636678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.636975 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.636975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.637018 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.637357 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:02.637361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:02.637624 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:02.639261 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.639261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.639262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.639423 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.639631 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.639632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.639632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.639691 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.640038 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:02.640039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:02.640091 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.640194 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.640269 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":75,"threads_started":1}
I20260924 19:27:02.640419 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.634711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:46277 (local address 127.18.229.190:40393)
0924 19:27:02.634747 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.634754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.635189 (+ 435us) server_negotiation.cc:407] Connection header received
0924 19:27:02.635236 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.635237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.635247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.635309 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.635636 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.636079 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.636740 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.636914 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.636961 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.637706 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.637708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.637708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.637715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.639166 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.639506 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.639507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.639508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.639559 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.639765 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.639765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.639766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.639957 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.640034 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.640227 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.640310 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for oryx/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab.
Entry for principal oryx/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab.
I20260924 19:27:02.660370 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40393
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:02.739184 27214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:02.739416 27214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:02.739511 27214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:02.741005 27214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:02.741078 27214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:02.741128 27214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:27:02.741169 27214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:27:02.742728 27214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.229.190:40393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:02.742969 27214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:02.743198 27214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:02.748047 27222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:02.748086 27219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.748277 27214 server_base.cc:1034] running on GCE node
W20260924 19:27:02.748083 27220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:02.748567 27214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:02.748791 27214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:02.749926 27214 hybrid_clock.cc:648] HybridClock initialized: now 1790278022749909 us; error 42 us; skew 500 ppm
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:02.752411 27214 init.cc:377] Logged in from keytab as oryx/127.18.229.131@KRBTEST.COM (short username oryx)
I20260924 19:27:02.752909 27214 webserver.cc:460] Webserver started at http://127.18.229.131:36575/ using document root <none> and password file <none>
I20260924 19:27:02.753132 27214 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:02.753181 27214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:02.753330 27214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:02.754354 27214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "c8901fed8cab4fe8aa317e8430986955"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "db32838c0d6556f3349354064b666061"
server_key_iv: "f379de064ff87f00224f61d848bccf5a"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.754707 27214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "c8901fed8cab4fe8aa317e8430986955"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "db32838c0d6556f3349354064b666061"
server_key_iv: "f379de064ff87f00224f61d848bccf5a"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.755980 27214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:02.756892 27229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.757104 27214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:02.757201 27214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "c8901fed8cab4fe8aa317e8430986955"
format_stamp: "Formatted at 2026-09-24 19:27:02 on dist-test-slave-r8cc"
server_key: "db32838c0d6556f3349354064b666061"
server_key_iv: "f379de064ff87f00224f61d848bccf5a"
server_key_version: "encryptionkey@0"
I20260924 19:27:02.757279 27214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:02.766572 27214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:02.768813 27214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:02.768922 27214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:02.769414 27214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:02.769480 27214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:02.769511 27214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:02.769531 27214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:02.776700 27214 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:45853
I20260924 19:27:02.777034 27214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:27:02.777408 27344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:45853 every 8 connection(s)
I20260924 19:27:02.778689 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.769769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:57319 (local address 127.18.229.190:40393)
0924 19:27:02.769812 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.769814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.770012 (+ 198us) server_negotiation.cc:407] Connection header received
0924 19:27:02.770486 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.770488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.770505 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.770576 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.771201 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.771774 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.772858 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.773056 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.773140 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.775741 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.775743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.775743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.775751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.777229 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.777746 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.777747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.777748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.777808 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.778055 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.778056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.778056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.778177 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.778254 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.778519 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.778597 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:27:02.778880 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.769756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.18.229.131:57319)
0924 19:27:02.769898 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.769932 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.770380 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.770645 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.770648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.770870 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:02.771106 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.771110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.771868 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.771870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.772755 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.772757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.773155 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.773156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.773316 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.773807 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:02.773815 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:02.775628 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:02.777387 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.777390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.777393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.777655 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.777873 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.777874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.777875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.777986 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.778244 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:02.778249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:02.778386 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.778496 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.778659 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260924 19:27:02.779330 27345 heartbeater.cc:347] Connected to a master server at 127.18.229.190:40393
I20260924 19:27:02.779399 27345 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:02.779575 27345 heartbeater.cc:511] Master 127.18.229.190:40393 requested a full tablet report, sending...
I20260924 19:27:02.779947 26861 ts_manager.cc:194] Registered new tserver with Master: c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:02.780381 26861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:57319
I20260924 19:27:02.784673 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 27214
I20260924 19:27:02.784811 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:27:02.785038 19350 external_mini_cluster.cc:1428] Setting key f118a9a6274f7cd91eb97e2c614c4a4b
I20260924 19:27:02.785655 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.780453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:47885 (local address 127.18.229.190:40393)
0924 19:27:02.780502 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.780504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.780589 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:27:02.780680 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.780681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.780690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.780733 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.781054 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.781517 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.782211 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.782415 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.782495 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.783251 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.783253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.783253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.783260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.784397 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.784838 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.784840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.784840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.784905 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.785089 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.785090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.785090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.785234 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.785300 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.785521 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.785598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:27:02.785660 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.780382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.18.229.131:47885)
0924 19:27:02.780526 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.780536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.780612 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.780746 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.780747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.780918 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:02.780976 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.780977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.781606 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.781607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.782113 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.782115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.782496 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.782497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.782559 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.782912 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:02.782916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:02.783168 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:02.784489 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.784489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.784490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.784739 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.784979 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:02.784980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:02.784980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:02.785031 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:02.785310 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:02.785311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:02.785358 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.785428 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.785524 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 19:27:02.785969 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:27:02.806599 26931 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:02.801392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.18.229.190:40393)
0924 19:27:02.801461 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.801463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.801474 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:27:02.801546 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.801547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.801564 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.801622 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.802192 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.802669 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.803433 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.803621 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.803729 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.806486 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47234: BlockingRecv error: recv got EOF from 127.0.0.1:47234 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 19:27:02 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:02.813447 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.807296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.18.229.190:40393)
0924 19:27:02.807352 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.807354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.807441 (+ 87us) server_negotiation.cc:407] Connection header received
0924 19:27:02.807478 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.807479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.807488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.807534 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:02.807946 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.808448 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.809072 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.809235 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.809280 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.811127 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:02.811128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:02.811129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:02.811136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:02.812332 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.812684 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.812685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.812686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.812744 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:02.812959 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:02.812960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:02.812961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:02.813064 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:02.813168 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.813283 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.813351 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:27:02.814816 26861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:27:02.816370 26861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:02.822050 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.818244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41455 (local address 127.0.0.1:46830)
0924 19:27:02.818394 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.818434 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.818499 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.818855 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.818856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.818859 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:02.818933 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.818935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.819827 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.819828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.820788 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.820790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.821740 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.821741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.821819 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.821833 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.821915 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.821973 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":54,"thread_start_us":49,"threads_started":1}
I20260924 19:27:02.822068 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.818324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.18.229.130:41455)
0924 19:27:02.818696 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.818701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.818709 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:02.818742 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.818744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.818761 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.818901 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:02.818993 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.819755 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.820880 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.821647 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.821740 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.821794 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.821938 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.821974 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":76,"threads_started":1}
I20260924 19:27:02.822351 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.818273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45853 (local address 127.0.0.1:39844)
0924 19:27:02.818476 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.818486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.818539 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.818837 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.818840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.818841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:02.818906 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.818908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.819967 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.819968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.821108 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.821111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.822035 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.822036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.822109 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.822210 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.822253 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.822291 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":58,"threads_started":1}
I20260924 19:27:02.822662 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.818459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.18.229.131:45853)
0924 19:27:02.818638 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.818644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.818654 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:27:02.818710 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.818712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.818729 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.818815 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:02.818986 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.819883 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.821206 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.821873 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.821953 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.822003 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.822512 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.822551 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260924 19:27:02.822800 27139 tablet_service.cc:1467] Processing CreateTablet for tablet d7a9d03a1faf4cc285698e83fe438d2f (DEFAULT_TABLE table=test-table [id=63a2ec33ce62492b8aeb6fb54ff56e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:02.823190 27139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7a9d03a1faf4cc285698e83fe438d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:02.823434 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.818318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40289 (local address 127.0.0.1:50624)
0924 19:27:02.818605 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.818614 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.818649 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.818868 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.818869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.818870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:02.818931 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.818933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.821003 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.821004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.822085 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.822088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.823212 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.823213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.823275 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.823279 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.823332 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.823371 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":4,"thread_start_us":38,"threads_started":1}
I20260924 19:27:02.823266 27279 tablet_service.cc:1467] Processing CreateTablet for tablet d7a9d03a1faf4cc285698e83fe438d2f (DEFAULT_TABLE table=test-table [id=63a2ec33ce62492b8aeb6fb54ff56e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:02.823686 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7a9d03a1faf4cc285698e83fe438d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:02.823748 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.818499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.18.229.129:40289)
0924 19:27:02.818642 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.818655 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.818668 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:27:02.818712 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.818715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.818730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.818810 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:02.819835 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.820922 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.822310 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.823034 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.823124 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.823175 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.823601 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.823644 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260924 19:27:02.824388 26999 tablet_service.cc:1467] Processing CreateTablet for tablet d7a9d03a1faf4cc285698e83fe438d2f (DEFAULT_TABLE table=test-table [id=63a2ec33ce62492b8aeb6fb54ff56e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:02.824740 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7a9d03a1faf4cc285698e83fe438d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:02.826678 27370 tablet_bootstrap.cc:492] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Bootstrap starting.
I20260924 19:27:02.827397 27370 tablet_bootstrap.cc:654] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:02.827421 27371 tablet_bootstrap.cc:492] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Bootstrap starting.
I20260924 19:27:02.827709 27370 log.cc:824] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:02.828024 27371 tablet_bootstrap.cc:654] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:02.828402 27371 log.cc:824] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:02.828439 27370 tablet_bootstrap.cc:492] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: No bootstrap required, opened a new log
I20260924 19:27:02.828516 27370 ts_tablet_manager.cc:1397] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:02.828732 27373 tablet_bootstrap.cc:492] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Bootstrap starting.
I20260924 19:27:02.829246 27373 tablet_bootstrap.cc:654] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:02.829527 27373 log.cc:824] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:02.830053 27370 raft_consensus.cc:348] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.830147 27371 tablet_bootstrap.cc:492] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: No bootstrap required, opened a new log
I20260924 19:27:02.830199 27370 raft_consensus.cc:374] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:02.830214 27371 ts_tablet_manager.cc:1397] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:27:02.830245 27370 raft_consensus.cc:729] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8901fed8cab4fe8aa317e8430986955, State: Initialized, Role: FOLLOWER
I20260924 19:27:02.830359 27370 consensus_queue.cc:260] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.830626 27370 ts_tablet_manager.cc:1428] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:27:02.830678 27345 heartbeater.cc:503] Master 127.18.229.190:40393 was elected leader, sending a full tablet report...
I20260924 19:27:02.832028 27371 raft_consensus.cc:348] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.832216 27371 raft_consensus.cc:374] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:02.832268 27371 raft_consensus.cc:729] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Initialized, Role: FOLLOWER
I20260924 19:27:02.832353 27371 consensus_queue.cc:260] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.832602 27371 ts_tablet_manager.cc:1428] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:27:02.832662 27373 tablet_bootstrap.cc:492] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: No bootstrap required, opened a new log
I20260924 19:27:02.832686 27205 heartbeater.cc:503] Master 127.18.229.190:40393 was elected leader, sending a full tablet report...
I20260924 19:27:02.832732 27373 ts_tablet_manager.cc:1397] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:27:02.834419 27373 raft_consensus.cc:348] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.834589 27373 raft_consensus.cc:374] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:02.834643 27373 raft_consensus.cc:729] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Initialized, Role: FOLLOWER
I20260924 19:27:02.834760 27373 consensus_queue.cc:260] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.835011 27373 ts_tablet_manager.cc:1428] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:27:02.835031 27065 heartbeater.cc:503] Master 127.18.229.190:40393 was elected leader, sending a full tablet report...
W20260924 19:27:02.880481 27206 tablet.cc:2369] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:27:02.957311 27378 raft_consensus.cc:482] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:02.957473 27378 raft_consensus.cc:504] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.957876 27378 leader_election.cc:288] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:02.960909 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.958082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41455 (local address 127.18.229.129:44399)
0924 19:27:02.958136 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.958147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.958192 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.958348 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.958349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.958351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:02.958429 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.958431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.959107 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.959108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.960075 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.960077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.960622 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.960624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.960718 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.960722 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.960779 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.960808 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":45}
I20260924 19:27:02.960923 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.958147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:44399 (local address 127.18.229.130:41455)
0924 19:27:02.958194 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.958196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.958207 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:27:02.958244 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.958245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.958255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.958320 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:02.958506 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.959034 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.960206 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.960555 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.960608 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.960615 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.960790 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.960854 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:27:02.960985 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.958259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:55445 (local address 127.18.229.131:45853)
0924 19:27:02.958327 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:02.958329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:02.958338 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:02.958400 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:02.958401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:02.958411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:02.958461 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:02.958643 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.959110 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.960157 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.960622 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.960693 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.960701 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:02.960876 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:27:02.960911 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:27:02.961055 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:02.958098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45853 (local address 127.18.229.129:55445)
0924 19:27:02.958278 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:02.958287 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:02.958331 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:02.958493 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:02.958494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:02.958496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:02.958575 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.958576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.959191 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.959192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.960056 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:02.960058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:02.960704 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:02.960705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:02.960786 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:02.960790 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:27:02.960846 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:27:02.960881 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":117,"threads_started":1}
I20260924 19:27:02.961373 27159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a9d03a1faf4cc285698e83fe438d2f" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" is_pre_election: true
I20260924 19:27:02.961382 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a9d03a1faf4cc285698e83fe438d2f" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955" is_pre_election: true
I20260924 19:27:02.961511 27159 raft_consensus.cc:2454] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 0.
I20260924 19:27:02.961519 27299 raft_consensus.cc:2454] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 0.
I20260924 19:27:02.961712 26951 leader_election.cc:302] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa; no voters:
I20260924 19:27:02.961846 27378 raft_consensus.cc:2781] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:02.961901 27378 raft_consensus.cc:482] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:02.961922 27378 raft_consensus.cc:3037] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:02.962859 27378 raft_consensus.cc:504] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.963001 27378 leader_election.cc:288] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 election: Requested vote from peers 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:02.963136 27159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a9d03a1faf4cc285698e83fe438d2f" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21"
I20260924 19:27:02.963146 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a9d03a1faf4cc285698e83fe438d2f" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955"
I20260924 19:27:02.963215 27299 raft_consensus.cc:3037] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:02.963217 27159 raft_consensus.cc:3037] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:02.964073 27159 raft_consensus.cc:2454] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 1.
I20260924 19:27:02.964073 27299 raft_consensus.cc:2454] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 1.
I20260924 19:27:02.964252 26951 leader_election.cc:302] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa; no voters:
I20260924 19:27:02.964356 27378 raft_consensus.cc:2781] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:02.964484 27378 raft_consensus.cc:686] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 1 LEADER]: Becoming Leader. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Running, Role: LEADER
I20260924 19:27:02.964588 27378 consensus_queue.cc:237] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } }
I20260924 19:27:02.965382 26861 catalog_manager.cc:5495] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa reported cstate change: term changed from 0 to 1, leader changed from <none> to bfad6ba329eb41659c4e2325d94919fa (127.18.229.129). New cstate: current_term: 1 leader_uuid: "bfad6ba329eb41659c4e2325d94919fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } health_report { overall_health: HEALTHY } } }
W20260924 19:27:02.982113 27066 tablet.cc:2369] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:27:03.003422 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.001273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.18.229.129:40289)
0924 19:27:03.001330 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.001332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.001340 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:03.001389 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.001390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.001405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.001463 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.001596 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.002046 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.002769 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.002947 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.002992 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.003132 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.003255 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.003324 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.003356 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.003371 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:27:03.005123 27299 raft_consensus.cc:1261] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Refusing update from remote peer bfad6ba329eb41659c4e2325d94919fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.005124 27159 raft_consensus.cc:1261] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Refusing update from remote peer bfad6ba329eb41659c4e2325d94919fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.005504 27380 consensus_queue.cc:1035] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.005640 27378 consensus_queue.cc:1035] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.007843 27383 mvcc.cc:205] Tried to move back new op lower bound from 7332978782226333696 to 7332978782063497216. Current Snapshot: MvccSnapshot[applied={T|T < 7332978782226333696}]
I20260924 19:27:03.008867 27384 mvcc.cc:205] Tried to move back new op lower bound from 7332978782226333696 to 7332978782063497216. Current Snapshot: MvccSnapshot[applied={T|T < 7332978782226333696 or (T in {7332978782226333696})}]
I20260924 19:27:03.009943 26862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.010032 26862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.010764 26862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Sending DeleteTablet for 3 replicas of tablet d7a9d03a1faf4cc285698e83fe438d2f
I20260924 19:27:03.011123 27279 tablet_service.cc:1514] Processing DeleteTablet for tablet d7a9d03a1faf4cc285698e83fe438d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39844
I20260924 19:27:03.011240 27139 tablet_service.cc:1514] Processing DeleteTablet for tablet d7a9d03a1faf4cc285698e83fe438d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:46830
I20260924 19:27:03.011323 26999 tablet_service.cc:1514] Processing DeleteTablet for tablet d7a9d03a1faf4cc285698e83fe438d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50624
I20260924 19:27:03.011462 27394 tablet_replica.cc:330] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: stopping tablet replica
I20260924 19:27:03.011535 27395 tablet_replica.cc:330] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: stopping tablet replica
I20260924 19:27:03.011622 27395 raft_consensus.cc:2229] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:27:03.011626 27394 raft_consensus.cc:2229] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.011720 27395 raft_consensus.cc:2258] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.011739 26862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:27:03.011863 27394 raft_consensus.cc:2258] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:27:03.011873 26862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:03.012199 27393 tablet_replica.cc:330] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: stopping tablet replica
I20260924 19:27:03.012301 27393 raft_consensus.cc:2229] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.012406 27393 raft_consensus.cc:2258] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.012567 27395 ts_tablet_manager.cc:1905] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.013270 27393 ts_tablet_manager.cc:1905] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.013325 27139 tablet_service.cc:1467] Processing CreateTablet for tablet 53bc688c985a42ac85260f7014dcbb8b (DEFAULT_TABLE table=test-table [id=7997e026b8ed464181c8c509cc138a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.013446 27139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bc688c985a42ac85260f7014dcbb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.014429 27395 ts_tablet_manager.cc:1918] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:03.014551 27395 log.cc:1196] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/d7a9d03a1faf4cc285698e83fe438d2f
I20260924 19:27:03.014864 27371 tablet_bootstrap.cc:492] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Bootstrap starting.
I20260924 19:27:03.014922 27279 tablet_service.cc:1467] Processing CreateTablet for tablet 53bc688c985a42ac85260f7014dcbb8b (DEFAULT_TABLE table=test-table [id=7997e026b8ed464181c8c509cc138a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.014904 27395 ts_tablet_manager.cc:1939] T d7a9d03a1faf4cc285698e83fe438d2f P bfad6ba329eb41659c4e2325d94919fa: Deleting consensus metadata
I20260924 19:27:03.015024 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bc688c985a42ac85260f7014dcbb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.015296 27371 tablet_bootstrap.cc:654] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.015328 27393 ts_tablet_manager.cc:1918] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:03.015398 27393 log.cc:1196] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/d7a9d03a1faf4cc285698e83fe438d2f
I20260924 19:27:03.015678 27393 ts_tablet_manager.cc:1939] T d7a9d03a1faf4cc285698e83fe438d2f P c8901fed8cab4fe8aa317e8430986955: Deleting consensus metadata
I20260924 19:27:03.015923 27371 tablet_bootstrap.cc:492] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: No bootstrap required, opened a new log
I20260924 19:27:03.015964 27371 ts_tablet_manager.cc:1397] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:03.016049 26849 catalog_manager.cc:4841] TS c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853): tablet d7a9d03a1faf4cc285698e83fe438d2f (table test-table [id=63a2ec33ce62492b8aeb6fb54ff56e72]) successfully deleted
I20260924 19:27:03.016111 27371 raft_consensus.cc:348] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.016232 27371 raft_consensus.cc:374] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.016255 27371 raft_consensus.cc:729] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.016305 27371 consensus_queue.cc:260] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.016340 26846 catalog_manager.cc:4841] TS bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289): tablet d7a9d03a1faf4cc285698e83fe438d2f (table test-table [id=63a2ec33ce62492b8aeb6fb54ff56e72]) successfully deleted
I20260924 19:27:03.016400 27371 ts_tablet_manager.cc:1428] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.016606 26999 tablet_service.cc:1467] Processing CreateTablet for tablet 53bc688c985a42ac85260f7014dcbb8b (DEFAULT_TABLE table=test-table [id=7997e026b8ed464181c8c509cc138a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.016739 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bc688c985a42ac85260f7014dcbb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.017549 27373 tablet_bootstrap.cc:492] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Bootstrap starting.
I20260924 19:27:03.017948 27373 tablet_bootstrap.cc:654] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.018277 27370 tablet_bootstrap.cc:492] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Bootstrap starting.
I20260924 19:27:03.018287 27394 ts_tablet_manager.cc:1905] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.018658 27370 tablet_bootstrap.cc:654] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.019080 27373 tablet_bootstrap.cc:492] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: No bootstrap required, opened a new log
I20260924 19:27:03.019134 27373 ts_tablet_manager.cc:1397] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:03.019301 27373 raft_consensus.cc:348] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.019373 27373 raft_consensus.cc:374] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.019392 27373 raft_consensus.cc:729] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.019430 27373 consensus_queue.cc:260] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.019521 27373 ts_tablet_manager.cc:1428] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.019955 27394 ts_tablet_manager.cc:1918] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:03.020026 27394 log.cc:1196] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/d7a9d03a1faf4cc285698e83fe438d2f
I20260924 19:27:03.020172 27370 tablet_bootstrap.cc:492] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: No bootstrap required, opened a new log
I20260924 19:27:03.020221 27370 ts_tablet_manager.cc:1397] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:03.020313 27394 ts_tablet_manager.cc:1939] T d7a9d03a1faf4cc285698e83fe438d2f P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting consensus metadata
I20260924 19:27:03.020385 27370 raft_consensus.cc:348] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.020457 27370 raft_consensus.cc:374] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.020478 27370 raft_consensus.cc:729] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8901fed8cab4fe8aa317e8430986955, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.020521 27370 consensus_queue.cc:260] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.020603 27370 ts_tablet_manager.cc:1428] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.020677 26847 catalog_manager.cc:4841] TS 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455): tablet d7a9d03a1faf4cc285698e83fe438d2f (table test-table [id=63a2ec33ce62492b8aeb6fb54ff56e72]) successfully deleted
W20260924 19:27:03.027676 27346 tablet.cc:2369] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:27:03.216285 27376 raft_consensus.cc:482] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:03.216384 27376 raft_consensus.cc:504] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.216761 27376 leader_election.cc:288] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289), 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455)
I20260924 19:27:03.219600 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.216960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40289 (local address 127.18.229.131:37469)
0924 19:27:03.217046 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.217058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.217121 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.217277 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.217278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.217280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:03.217356 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.217358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.218014 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.218016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.218805 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.218807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.219330 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.219331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.219421 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.219424 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.219476 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.219506 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":21}
I20260924 19:27:03.219601 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.217002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:37469 (local address 127.18.229.129:40289)
0924 19:27:03.217058 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.217061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.217124 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:27:03.217191 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.217193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.217206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.217263 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:03.217429 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.217939 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.218891 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.219258 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.219313 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.219320 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.219489 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.219534 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:27:03.219666 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.217073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:43905 (local address 127.18.229.130:41455)
0924 19:27:03.217111 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.217114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.217234 (+ 120us) server_negotiation.cc:407] Connection header received
0924 19:27:03.217279 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.217281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.217289 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.217348 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:03.217538 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.218008 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.218937 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.219275 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.219373 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.219381 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.219561 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.219599 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:27:03.219887 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.216962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41455 (local address 127.18.229.131:43905)
0924 19:27:03.217144 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.217154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.217221 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.217358 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.217358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.217360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:03.217466 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.217468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.218085 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.218085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.218815 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.218816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.219411 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.219412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.219481 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.219485 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.219538 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.219571 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":55,"thread_start_us":57,"threads_started":1}
I20260924 19:27:03.219908 27019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc688c985a42ac85260f7014dcbb8b" candidate_uuid: "c8901fed8cab4fe8aa317e8430986955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad6ba329eb41659c4e2325d94919fa" is_pre_election: true
I20260924 19:27:03.220054 27019 raft_consensus.cc:2454] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8901fed8cab4fe8aa317e8430986955 in term 0.
I20260924 19:27:03.220139 27159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc688c985a42ac85260f7014dcbb8b" candidate_uuid: "c8901fed8cab4fe8aa317e8430986955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" is_pre_election: true
I20260924 19:27:03.220232 27159 raft_consensus.cc:2454] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8901fed8cab4fe8aa317e8430986955 in term 0.
I20260924 19:27:03.220285 27230 leader_election.cc:302] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfad6ba329eb41659c4e2325d94919fa, c8901fed8cab4fe8aa317e8430986955; no voters:
I20260924 19:27:03.220384 27376 raft_consensus.cc:2781] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:03.220420 27376 raft_consensus.cc:482] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:03.220439 27376 raft_consensus.cc:3037] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.221052 27376 raft_consensus.cc:504] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.221208 27376 leader_election.cc:288] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [CANDIDATE]: Term 1 election: Requested vote from peers bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289), 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455)
I20260924 19:27:03.221341 27019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc688c985a42ac85260f7014dcbb8b" candidate_uuid: "c8901fed8cab4fe8aa317e8430986955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad6ba329eb41659c4e2325d94919fa"
I20260924 19:27:03.221408 27019 raft_consensus.cc:3037] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.221357 27159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc688c985a42ac85260f7014dcbb8b" candidate_uuid: "c8901fed8cab4fe8aa317e8430986955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21"
I20260924 19:27:03.221455 27159 raft_consensus.cc:3037] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.222019 27019 raft_consensus.cc:2454] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8901fed8cab4fe8aa317e8430986955 in term 1.
I20260924 19:27:03.222131 27159 raft_consensus.cc:2454] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8901fed8cab4fe8aa317e8430986955 in term 1.
I20260924 19:27:03.222177 27230 leader_election.cc:302] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfad6ba329eb41659c4e2325d94919fa, c8901fed8cab4fe8aa317e8430986955; no voters:
I20260924 19:27:03.222283 27376 raft_consensus.cc:2781] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:03.222445 27376 raft_consensus.cc:686] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 1 LEADER]: Becoming Leader. State: Replica: c8901fed8cab4fe8aa317e8430986955, State: Running, Role: LEADER
I20260924 19:27:03.222523 27376 consensus_queue.cc:237] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.223289 26861 catalog_manager.cc:5495] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8901fed8cab4fe8aa317e8430986955 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "c8901fed8cab4fe8aa317e8430986955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } health_report { overall_health: HEALTHY } } }
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278022, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.190@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:03.263419 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.256822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40393 (local address 127.0.0.1:47258)
0924 19:27:03.256991 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.257001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.257066 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.257192 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.257193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.257477 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:03.257527 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.257529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.258129 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.258129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.258664 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.258665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.258958 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.258959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.258997 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.259528 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:03.259533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:03.260953 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:03.262335 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:03.262342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:03.262344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:03.262522 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:03.262749 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:03.262750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:03.262751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:03.262805 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:03.263051 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:03.263057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:03.263100 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.263187 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.263296 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260924 19:27:03.263425 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.256892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.18.229.190:40393)
0924 19:27:03.256957 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.256960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.257064 (+ 104us) server_negotiation.cc:407] Connection header received
0924 19:27:03.257107 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.257107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.257116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.257170 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:03.257586 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.258063 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.258728 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.258898 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.258946 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.261041 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:03.261042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:03.261043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:03.261049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:03.262231 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:03.262607 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:03.262608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:03.262609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:03.262685 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:03.262880 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:03.262881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:03.262882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:03.262985 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:03.263060 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.263223 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.263346 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":41}
I20260924 19:27:03.265211 26861 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:47258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:27:03.265437 26861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:03.266891 26999 tablet_service.cc:1467] Processing CreateTablet for tablet 579c349e2d36406d8cb6600323db99b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=076348f2ee004aac8520a6818ba647a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:27:03.266903 27139 tablet_service.cc:1467] Processing CreateTablet for tablet 579c349e2d36406d8cb6600323db99b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=076348f2ee004aac8520a6818ba647a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:27:03.266942 27279 tablet_service.cc:1467] Processing CreateTablet for tablet 579c349e2d36406d8cb6600323db99b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=076348f2ee004aac8520a6818ba647a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:27:03.267045 27139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579c349e2d36406d8cb6600323db99b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.267045 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579c349e2d36406d8cb6600323db99b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.267099 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579c349e2d36406d8cb6600323db99b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.268203 27370 tablet_bootstrap.cc:492] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: Bootstrap starting.
I20260924 19:27:03.268493 27373 tablet_bootstrap.cc:492] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: Bootstrap starting.
I20260924 19:27:03.268515 27371 tablet_bootstrap.cc:492] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Bootstrap starting.
I20260924 19:27:03.268641 27370 tablet_bootstrap.cc:654] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.268826 27373 tablet_bootstrap.cc:654] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.268919 27371 tablet_bootstrap.cc:654] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.269390 27370 tablet_bootstrap.cc:492] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: No bootstrap required, opened a new log
I20260924 19:27:03.269443 27370 ts_tablet_manager.cc:1397] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:03.269454 27371 tablet_bootstrap.cc:492] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: No bootstrap required, opened a new log
I20260924 19:27:03.269495 27371 ts_tablet_manager.cc:1397] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:27:03.269588 27370 raft_consensus.cc:348] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.269672 27370 raft_consensus.cc:374] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.269642 27371 raft_consensus.cc:348] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.269694 27370 raft_consensus.cc:729] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8901fed8cab4fe8aa317e8430986955, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.269701 27371 raft_consensus.cc:374] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.269723 27371 raft_consensus.cc:729] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.269745 27370 consensus_queue.cc:260] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.269773 27371 consensus_queue.cc:260] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.269848 27370 ts_tablet_manager.cc:1428] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:03.269861 27371 ts_tablet_manager.cc:1428] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:03.269974 27377 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.269965 27376 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.270041 27376 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.270040 27377 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.270591 27373 tablet_bootstrap.cc:492] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: No bootstrap required, opened a new log
I20260924 19:27:03.270653 27373 ts_tablet_manager.cc:1397] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:03.270779 27373 raft_consensus.cc:348] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.270850 27373 raft_consensus.cc:374] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.270874 27373 raft_consensus.cc:729] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.270925 27373 consensus_queue.cc:260] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.271018 27373 ts_tablet_manager.cc:1428] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.271066 27390 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.271201 27390 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.390089 27377 raft_consensus.cc:482] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:03.390195 27377 raft_consensus.cc:504] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.390575 27377 leader_election.cc:288] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:03.393832 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.390823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:58179 (local address 127.18.229.131:45853)
0924 19:27:03.390891 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.390893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.390947 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:27:03.391038 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.391039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.391048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.391098 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:03.391276 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.391767 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.392968 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.393490 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.393573 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.393583 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.393736 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.393772 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:27:03.393852 27411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.390763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45853 (local address 127.18.229.130:58179)
0924 19:27:03.390885 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.390897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.390964 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.391126 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.391127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.391129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:03.391201 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.391204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.391853 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.391854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.392880 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.392882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.393570 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.393571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.393642 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.393648 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.393718 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.393758 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":64,"thread_start_us":59,"threads_started":1}
I20260924 19:27:03.393916 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.390826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:40149 (local address 127.18.229.129:40289)
0924 19:27:03.390900 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.390902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.390947 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:27:03.391038 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.391039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.391048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.391128 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:03.391321 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.391769 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.392965 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.393525 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.393631 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.393640 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.393801 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.393842 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:27:03.393965 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.390770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40289 (local address 127.18.229.130:40149)
0924 19:27:03.390885 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.390897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.390966 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.391158 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.391159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.391161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:03.391231 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.391232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.391896 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.391897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.392867 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.392869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.393635 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.393636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.393722 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.393725 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.393795 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.393837 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":12,"thread_start_us":60,"threads_started":1}
I20260924 19:27:03.394049 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c349e2d36406d8cb6600323db99b5" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955" is_pre_election: true
I20260924 19:27:03.394126 27299 raft_consensus.cc:2454] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 0.
I20260924 19:27:03.394309 27093 leader_election.cc:302] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, c8901fed8cab4fe8aa317e8430986955; no voters:
I20260924 19:27:03.394338 27019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c349e2d36406d8cb6600323db99b5" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad6ba329eb41659c4e2325d94919fa" is_pre_election: true
I20260924 19:27:03.394412 27019 raft_consensus.cc:2454] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 0.
I20260924 19:27:03.394456 27377 raft_consensus.cc:2781] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:03.394512 27377 raft_consensus.cc:482] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:03.394536 27377 raft_consensus.cc:3037] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.395192 27377 raft_consensus.cc:504] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.395327 27377 leader_election.cc:288] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 election: Requested vote from peers bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:03.395462 27019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c349e2d36406d8cb6600323db99b5" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad6ba329eb41659c4e2325d94919fa"
I20260924 19:27:03.395491 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c349e2d36406d8cb6600323db99b5" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955"
I20260924 19:27:03.395541 27019 raft_consensus.cc:3037] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.395545 27299 raft_consensus.cc:3037] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.396279 27299 raft_consensus.cc:2454] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 1.
I20260924 19:27:03.396279 27019 raft_consensus.cc:2454] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 1.
I20260924 19:27:03.396436 27090 leader_election.cc:302] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa; no voters:
I20260924 19:27:03.396534 27377 raft_consensus.cc:2781] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:03.396639 27377 raft_consensus.cc:686] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 LEADER]: Becoming Leader. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Running, Role: LEADER
I20260924 19:27:03.396741 27377 consensus_queue.cc:237] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.397073 27413 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: TxnStatusTablet state changed. Reason: New leader 08d4271eb2ae4b66ae7ba30cfb7b7c21. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.397200 27413 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:27:03.397353 27415 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:27:03.397459 27415 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:27:03.397444 26860 catalog_manager.cc:5495] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } health_report { overall_health: UNKNOWN } } }
I20260924 19:27:03.417090 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.414432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.18.229.130:41455)
0924 19:27:03.414486 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.414488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.414495 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:27:03.414521 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.414522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.414550 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.414613 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.414763 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.415233 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.416353 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.416530 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.416582 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.416773 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.416868 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.416925 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.417003 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.417020 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:27:03.417132 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.414176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41455 (local address 127.0.0.1:46832)
0924 19:27:03.414275 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.414288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.414342 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.414645 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.414647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.414648 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:27:03.414699 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.414701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.415311 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.415311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.416277 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.416280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.416605 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.416606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.416670 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.416688 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.416936 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.416945 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.417001 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.417051 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:27:03.418238 27299 raft_consensus.cc:1261] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Refusing update from remote peer 08d4271eb2ae4b66ae7ba30cfb7b7c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.418314 27019 raft_consensus.cc:1261] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Refusing update from remote peer 08d4271eb2ae4b66ae7ba30cfb7b7c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.418488 27413 consensus_queue.cc:1035] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.418632 27377 consensus_queue.cc:1035] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.419269 27390 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: TxnStatusTablet state changed. Reason: New leader 08d4271eb2ae4b66ae7ba30cfb7b7c21. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.419355 27390 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.419593 27376 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: TxnStatusTablet state changed. Reason: New leader 08d4271eb2ae4b66ae7ba30cfb7b7c21. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.419689 27376 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.420264 27413 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.420341 27413 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:27:03.420476 27390 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.420472 27377 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.420564 27390 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P bfad6ba329eb41659c4e2325d94919fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.420568 27377 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:27:03.421332 27385 mvcc.cc:205] Tried to move back new op lower bound from 7332978783919149056 to 7332978783833567232. Current Snapshot: MvccSnapshot[applied={T|T < 7332978783919149056}]
I20260924 19:27:03.421413 27376 tablet_replica.cc:439] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } } }
I20260924 19:27:03.421496 27376 tablet_replica.cc:442] T 579c349e2d36406d8cb6600323db99b5 P c8901fed8cab4fe8aa317e8430986955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:27:03.424342 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.422092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.18.229.131:45853)
0924 19:27:03.422140 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.422142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.422160 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:27:03.422221 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.422221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.422239 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.422306 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.422408 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.422841 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.423549 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.423765 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.423859 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.424017 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.424144 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.424218 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.424257 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.424275 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:27:03.429068 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.426490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41455 (local address 127.18.229.131:40705)
0924 19:27:03.426608 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.426616 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.426657 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.426766 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.426767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.426768 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:27:03.426811 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.426812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.427624 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.427624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.428278 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.428280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.428723 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.428723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.428765 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.428776 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.428947 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.428948 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.428982 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.429011 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260924 19:27:03.429067 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.426605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:40705 (local address 127.18.229.130:41455)
0924 19:27:03.426640 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.426642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.426651 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:03.426698 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.426699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.426707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.426765 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.426845 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.427557 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.428349 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.428649 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.428719 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.428846 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.428877 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.428944 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.428984 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.429002 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:27:03.431576 27019 raft_consensus.cc:1261] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Refusing update from remote peer c8901fed8cab4fe8aa317e8430986955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.431622 27159 raft_consensus.cc:1261] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Refusing update from remote peer c8901fed8cab4fe8aa317e8430986955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.431787 27376 consensus_queue.cc:1035] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.431910 27400 consensus_queue.cc:1035] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.440389 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.437578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:37475 (local address 127.18.229.131:45853)
0924 19:27:03.437614 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.437616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.437776 (+ 160us) server_negotiation.cc:407] Connection header received
0924 19:27:03.437835 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.437835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.437845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.437890 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.438075 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.438793 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.439586 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.439888 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.439961 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.440188 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.440220 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.440260 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.440305 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.440323 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:27:03.440402 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.437541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45853 (local address 127.18.229.130:37475)
0924 19:27:03.437700 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.437717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.437786 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.437927 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.437928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.437929 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:27:03.437985 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.437986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.438872 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.438872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.439512 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.439514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.439973 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.439974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.440055 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.440071 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.440271 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.440272 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.440302 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.440335 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260924 19:27:03.447090 26862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.447189 26862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.447767 26862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Sending DeleteTablet for 3 replicas of tablet 53bc688c985a42ac85260f7014dcbb8b
I20260924 19:27:03.447992 27139 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc688c985a42ac85260f7014dcbb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:46830
I20260924 19:27:03.448074 27394 tablet_replica.cc:330] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: stopping tablet replica
I20260924 19:27:03.448150 27394 raft_consensus.cc:2229] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.448191 27394 raft_consensus.cc:2258] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.448164 26999 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc688c985a42ac85260f7014dcbb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50624
I20260924 19:27:03.448259 27395 tablet_replica.cc:330] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: stopping tablet replica
I20260924 19:27:03.448316 27395 raft_consensus.cc:2229] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.448349 27395 raft_consensus.cc:2258] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.448402 27279 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc688c985a42ac85260f7014dcbb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39844
I20260924 19:27:03.448499 27393 tablet_replica.cc:330] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: stopping tablet replica
I20260924 19:27:03.448544 27393 raft_consensus.cc:2229] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:27:03.448614 27393 raft_consensus.cc:2258] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.448882 27393 ts_tablet_manager.cc:1905] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.448953 27395 ts_tablet_manager.cc:1905] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.450480 27393 ts_tablet_manager.cc:1918] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:27:03.450485 27395 ts_tablet_manager.cc:1918] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:27:03.450531 27393 log.cc:1196] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/53bc688c985a42ac85260f7014dcbb8b
I20260924 19:27:03.450533 27395 log.cc:1196] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/53bc688c985a42ac85260f7014dcbb8b
I20260924 19:27:03.450754 27395 ts_tablet_manager.cc:1939] T 53bc688c985a42ac85260f7014dcbb8b P bfad6ba329eb41659c4e2325d94919fa: Deleting consensus metadata
I20260924 19:27:03.450766 27393 ts_tablet_manager.cc:1939] T 53bc688c985a42ac85260f7014dcbb8b P c8901fed8cab4fe8aa317e8430986955: Deleting consensus metadata
I20260924 19:27:03.451130 26849 catalog_manager.cc:4841] TS c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853): tablet 53bc688c985a42ac85260f7014dcbb8b (table test-table [id=7997e026b8ed464181c8c509cc138a21]) successfully deleted
I20260924 19:27:03.451344 27394 ts_tablet_manager.cc:1905] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.452800 27394 ts_tablet_manager.cc:1918] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:27:03.452847 27394 log.cc:1196] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/53bc688c985a42ac85260f7014dcbb8b
I20260924 19:27:03.453043 27394 ts_tablet_manager.cc:1939] T 53bc688c985a42ac85260f7014dcbb8b P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting consensus metadata
I20260924 19:27:03.453478 26847 catalog_manager.cc:4841] TS 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455): tablet 53bc688c985a42ac85260f7014dcbb8b (table test-table [id=7997e026b8ed464181c8c509cc138a21]) successfully deleted
I20260924 19:27:03.453650 26846 catalog_manager.cc:4841] TS bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289): tablet 53bc688c985a42ac85260f7014dcbb8b (table test-table [id=7997e026b8ed464181c8c509cc138a21]) successfully deleted
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[26817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:27:03.484001 26931 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:03.477280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.18.229.190:40393)
0924 19:27:03.477330 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.477332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.477431 (+ 99us) server_negotiation.cc:407] Connection header received
0924 19:27:03.477494 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.477495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.477507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.477583 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:03.478295 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.479067 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.480216 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.480511 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.480594 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.483936 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47268: BlockingRecv error: recv got EOF from 127.0.0.1:47268 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[26817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM
I20260924 19:27:03.494063 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.485452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.18.229.190:40393)
0924 19:27:03.485489 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.485491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.485551 (+ 60us) server_negotiation.cc:407] Connection header received
0924 19:27:03.485612 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.485613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.485622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.485705 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:03.486227 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.486925 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.487945 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.488291 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.488359 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.490997 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:03.490998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:03.490999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:03.491006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:03.492746 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:03.493160 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:03.493161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:03.493162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:03.493231 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:03.493451 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:03.493452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:03.493453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:03.493628 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:03.493761 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.493885 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.493968 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:27:03.495900 26862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:27:03.496071 26862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:03.497408 27279 tablet_service.cc:1467] Processing CreateTablet for tablet 144d2d737013444ab1a2a92f61ba45d5 (DEFAULT_TABLE table=test-table [id=b1f5a99bb7c54e628f4f6d75ad5e1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.497416 27139 tablet_service.cc:1467] Processing CreateTablet for tablet 144d2d737013444ab1a2a92f61ba45d5 (DEFAULT_TABLE table=test-table [id=b1f5a99bb7c54e628f4f6d75ad5e1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.497545 27139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144d2d737013444ab1a2a92f61ba45d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.497545 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144d2d737013444ab1a2a92f61ba45d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.497524 26999 tablet_service.cc:1467] Processing CreateTablet for tablet 144d2d737013444ab1a2a92f61ba45d5 (DEFAULT_TABLE table=test-table [id=b1f5a99bb7c54e628f4f6d75ad5e1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.497625 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144d2d737013444ab1a2a92f61ba45d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.498868 27370 tablet_bootstrap.cc:492] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Bootstrap starting.
I20260924 19:27:03.498889 27371 tablet_bootstrap.cc:492] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Bootstrap starting.
I20260924 19:27:03.499274 27370 tablet_bootstrap.cc:654] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.499318 27371 tablet_bootstrap.cc:654] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.499577 27373 tablet_bootstrap.cc:492] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Bootstrap starting.
I20260924 19:27:03.499856 27373 tablet_bootstrap.cc:654] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.500424 27373 tablet_bootstrap.cc:492] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: No bootstrap required, opened a new log
I20260924 19:27:03.500473 27373 ts_tablet_manager.cc:1397] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:03.500581 27373 raft_consensus.cc:348] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.500645 27373 raft_consensus.cc:374] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.500641 27370 tablet_bootstrap.cc:492] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: No bootstrap required, opened a new log
I20260924 19:27:03.500667 27373 raft_consensus.cc:729] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.500701 27370 ts_tablet_manager.cc:1397] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:03.500710 27373 consensus_queue.cc:260] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.500833 27373 ts_tablet_manager.cc:1428] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.500836 27370 raft_consensus.cc:348] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.500896 27370 raft_consensus.cc:374] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.500916 27370 raft_consensus.cc:729] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8901fed8cab4fe8aa317e8430986955, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.500958 27370 consensus_queue.cc:260] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.501040 27370 ts_tablet_manager.cc:1428] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:03.504483 27371 tablet_bootstrap.cc:492] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: No bootstrap required, opened a new log
I20260924 19:27:03.504532 27371 ts_tablet_manager.cc:1397] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260924 19:27:03.504670 27371 raft_consensus.cc:348] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.504734 27371 raft_consensus.cc:374] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.504755 27371 raft_consensus.cc:729] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.504812 27371 consensus_queue.cc:260] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.504894 27371 ts_tablet_manager.cc:1428] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.611564 27390 raft_consensus.cc:482] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:03.611677 27390 raft_consensus.cc:504] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.611896 27390 leader_election.cc:288] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:03.612150 27159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144d2d737013444ab1a2a92f61ba45d5" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" is_pre_election: true
I20260924 19:27:03.612161 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144d2d737013444ab1a2a92f61ba45d5" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955" is_pre_election: true
I20260924 19:27:03.612265 27159 raft_consensus.cc:2454] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 0.
I20260924 19:27:03.612294 27299 raft_consensus.cc:2454] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 0.
I20260924 19:27:03.612429 26951 leader_election.cc:302] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa; no voters:
I20260924 19:27:03.612546 27390 raft_consensus.cc:2781] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:03.612629 27390 raft_consensus.cc:482] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:03.612666 27390 raft_consensus.cc:3037] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.613335 27390 raft_consensus.cc:504] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.613480 27390 leader_election.cc:288] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 election: Requested vote from peers 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:03.613667 27159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144d2d737013444ab1a2a92f61ba45d5" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21"
I20260924 19:27:03.613673 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144d2d737013444ab1a2a92f61ba45d5" candidate_uuid: "bfad6ba329eb41659c4e2325d94919fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955"
I20260924 19:27:03.613746 27159 raft_consensus.cc:3037] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.613754 27299 raft_consensus.cc:3037] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.614499 27299 raft_consensus.cc:2454] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 1.
I20260924 19:27:03.614487 27159 raft_consensus.cc:2454] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfad6ba329eb41659c4e2325d94919fa in term 1.
I20260924 19:27:03.614699 26951 leader_election.cc:302] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa; no voters:
I20260924 19:27:03.614802 27390 raft_consensus.cc:2781] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:03.614917 27390 raft_consensus.cc:686] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 1 LEADER]: Becoming Leader. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Running, Role: LEADER
I20260924 19:27:03.614984 27390 consensus_queue.cc:237] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.615577 26860 catalog_manager.cc:5495] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa reported cstate change: term changed from 0 to 1, leader changed from <none> to bfad6ba329eb41659c4e2325d94919fa (127.18.229.129). New cstate: current_term: 1 leader_uuid: "bfad6ba329eb41659c4e2325d94919fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } health_report { overall_health: UNKNOWN } } }
I20260924 19:27:03.648859 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.646467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.18.229.129:40289)
0924 19:27:03.646542 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.646544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.646553 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:03.646601 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.646602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.646615 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.646683 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.646825 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.647320 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.648064 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.648328 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.648414 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.648574 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.648611 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.648685 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.648747 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.648768 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:27:03.649629 27159 raft_consensus.cc:1261] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Refusing update from remote peer bfad6ba329eb41659c4e2325d94919fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.649691 27299 raft_consensus.cc:1261] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Refusing update from remote peer bfad6ba329eb41659c4e2325d94919fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.649890 27446 consensus_queue.cc:1035] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.649993 27390 consensus_queue.cc:1035] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.652367 26860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.652477 26860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.653064 26860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Sending DeleteTablet for 3 replicas of tablet 144d2d737013444ab1a2a92f61ba45d5
I20260924 19:27:03.653283 26999 tablet_service.cc:1514] Processing DeleteTablet for tablet 144d2d737013444ab1a2a92f61ba45d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50624
I20260924 19:27:03.653321 27139 tablet_service.cc:1514] Processing DeleteTablet for tablet 144d2d737013444ab1a2a92f61ba45d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:46830
I20260924 19:27:03.653326 27279 tablet_service.cc:1514] Processing DeleteTablet for tablet 144d2d737013444ab1a2a92f61ba45d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39844
I20260924 19:27:03.653366 27395 tablet_replica.cc:330] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: stopping tablet replica
I20260924 19:27:03.653394 27394 tablet_replica.cc:330] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: stopping tablet replica
I20260924 19:27:03.653414 27395 raft_consensus.cc:2229] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:27:03.653414 27393 tablet_replica.cc:330] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: stopping tablet replica
I20260924 19:27:03.653439 27394 raft_consensus.cc:2229] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.653470 27395 raft_consensus.cc:2258] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.653468 27393 raft_consensus.cc:2229] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.653478 27394 raft_consensus.cc:2258] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.653518 27393 raft_consensus.cc:2258] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.653573 26860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:27:03.653721 26860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:03.653741 27395 ts_tablet_manager.cc:1905] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.653955 27394 ts_tablet_manager.cc:1905] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.655048 27139 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac9a22a4fa6403eab80331cf0f5a523 (DEFAULT_TABLE table=test-table [id=624988c3e8e645c7931dab156b5fe2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.655135 27279 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac9a22a4fa6403eab80331cf0f5a523 (DEFAULT_TABLE table=test-table [id=624988c3e8e645c7931dab156b5fe2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.655174 27139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac9a22a4fa6403eab80331cf0f5a523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.655241 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac9a22a4fa6403eab80331cf0f5a523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.655387 27395 ts_tablet_manager.cc:1918] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:03.655436 27395 log.cc:1196] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/144d2d737013444ab1a2a92f61ba45d5
I20260924 19:27:03.655659 27395 ts_tablet_manager.cc:1939] T 144d2d737013444ab1a2a92f61ba45d5 P bfad6ba329eb41659c4e2325d94919fa: Deleting consensus metadata
I20260924 19:27:03.655710 27394 ts_tablet_manager.cc:1918] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:03.655754 27394 log.cc:1196] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/144d2d737013444ab1a2a92f61ba45d5
I20260924 19:27:03.655961 27394 ts_tablet_manager.cc:1939] T 144d2d737013444ab1a2a92f61ba45d5 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting consensus metadata
I20260924 19:27:03.656260 26999 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac9a22a4fa6403eab80331cf0f5a523 (DEFAULT_TABLE table=test-table [id=624988c3e8e645c7931dab156b5fe2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:03.656294 26847 catalog_manager.cc:4841] TS 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455): tablet 144d2d737013444ab1a2a92f61ba45d5 (table test-table [id=b1f5a99bb7c54e628f4f6d75ad5e1f1b]) successfully deleted
I20260924 19:27:03.656354 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac9a22a4fa6403eab80331cf0f5a523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:03.656415 27371 tablet_bootstrap.cc:492] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Bootstrap starting.
I20260924 19:27:03.656556 27370 tablet_bootstrap.cc:492] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Bootstrap starting.
I20260924 19:27:03.656692 27371 tablet_bootstrap.cc:654] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.656821 26846 catalog_manager.cc:4841] TS bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289): tablet 144d2d737013444ab1a2a92f61ba45d5 (table test-table [id=b1f5a99bb7c54e628f4f6d75ad5e1f1b]) successfully deleted
I20260924 19:27:03.656880 27370 tablet_bootstrap.cc:654] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.657436 27370 tablet_bootstrap.cc:492] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: No bootstrap required, opened a new log
I20260924 19:27:03.657456 27371 tablet_bootstrap.cc:492] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: No bootstrap required, opened a new log
I20260924 19:27:03.657503 27371 ts_tablet_manager.cc:1397] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:27:03.657501 27370 ts_tablet_manager.cc:1397] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:03.657588 27373 tablet_bootstrap.cc:492] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa: Bootstrap starting.
I20260924 19:27:03.657617 27371 raft_consensus.cc:348] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.657634 27370 raft_consensus.cc:348] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.657682 27371 raft_consensus.cc:374] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.657706 27371 raft_consensus.cc:729] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.657706 27370 raft_consensus.cc:374] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.657730 27370 raft_consensus.cc:729] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8901fed8cab4fe8aa317e8430986955, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.657749 27371 consensus_queue.cc:260] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.657778 27370 consensus_queue.cc:260] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.657845 27371 ts_tablet_manager.cc:1428] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.657869 27370 ts_tablet_manager.cc:1428] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.657876 27373 tablet_bootstrap.cc:654] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:03.658421 27373 tablet_bootstrap.cc:492] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa: No bootstrap required, opened a new log
I20260924 19:27:03.658437 27393 ts_tablet_manager.cc:1905] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.658468 27373 ts_tablet_manager.cc:1397] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:27:03.658561 27373 raft_consensus.cc:348] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.658622 27373 raft_consensus.cc:374] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:03.658640 27373 raft_consensus.cc:729] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad6ba329eb41659c4e2325d94919fa, State: Initialized, Role: FOLLOWER
I20260924 19:27:03.658686 27373 consensus_queue.cc:260] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.658762 27373 ts_tablet_manager.cc:1428] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:03.659677 27393 ts_tablet_manager.cc:1918] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:03.659737 27393 log.cc:1196] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/144d2d737013444ab1a2a92f61ba45d5
I20260924 19:27:03.659955 27393 ts_tablet_manager.cc:1939] T 144d2d737013444ab1a2a92f61ba45d5 P c8901fed8cab4fe8aa317e8430986955: Deleting consensus metadata
I20260924 19:27:03.660243 26849 catalog_manager.cc:4841] TS c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853): tablet 144d2d737013444ab1a2a92f61ba45d5 (table test-table [id=b1f5a99bb7c54e628f4f6d75ad5e1f1b]) successfully deleted
I20260924 19:27:03.727362 27413 raft_consensus.cc:482] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:03.727468 27413 raft_consensus.cc:504] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.727651 27413 leader_election.cc:288] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:03.727852 27019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9a22a4fa6403eab80331cf0f5a523" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad6ba329eb41659c4e2325d94919fa" is_pre_election: true
I20260924 19:27:03.727864 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9a22a4fa6403eab80331cf0f5a523" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955" is_pre_election: true
I20260924 19:27:03.727960 27019 raft_consensus.cc:2454] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 0.
I20260924 19:27:03.727954 27299 raft_consensus.cc:2454] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 0.
I20260924 19:27:03.728161 27090 leader_election.cc:302] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa; no voters:
I20260924 19:27:03.728259 27413 raft_consensus.cc:2781] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:03.728307 27413 raft_consensus.cc:482] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:03.728335 27413 raft_consensus.cc:3037] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.729074 27413 raft_consensus.cc:504] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.729218 27413 leader_election.cc:288] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 election: Requested vote from peers bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289), c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853)
I20260924 19:27:03.729377 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9a22a4fa6403eab80331cf0f5a523" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8901fed8cab4fe8aa317e8430986955"
I20260924 19:27:03.729404 27019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac9a22a4fa6403eab80331cf0f5a523" candidate_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad6ba329eb41659c4e2325d94919fa"
I20260924 19:27:03.729457 27019 raft_consensus.cc:3037] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.729466 27299 raft_consensus.cc:3037] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:03.730057 27019 raft_consensus.cc:2454] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 1.
I20260924 19:27:03.730059 27299 raft_consensus.cc:2454] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d4271eb2ae4b66ae7ba30cfb7b7c21 in term 1.
I20260924 19:27:03.730237 27090 leader_election.cc:302] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d4271eb2ae4b66ae7ba30cfb7b7c21, bfad6ba329eb41659c4e2325d94919fa, c8901fed8cab4fe8aa317e8430986955; no voters:
I20260924 19:27:03.730348 27413 raft_consensus.cc:2781] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:03.730417 27413 raft_consensus.cc:686] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 LEADER]: Becoming Leader. State: Replica: 08d4271eb2ae4b66ae7ba30cfb7b7c21, State: Running, Role: LEADER
I20260924 19:27:03.730465 27413 consensus_queue.cc:237] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } }
I20260924 19:27:03.731038 26860 catalog_manager.cc:5495] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d4271eb2ae4b66ae7ba30cfb7b7c21" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 } health_report { overall_health: UNKNOWN } } }
I20260924 19:27:03.752465 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.750263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.18.229.130:41455)
0924 19:27:03.750315 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.750316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.750339 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:27:03.750412 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.750413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.750424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.750463 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.750568 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.751020 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.751766 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.751974 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.752047 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.752202 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.752236 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.752303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.752363 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.752382 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:27:03.756284 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.754497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:56297 (local address 127.18.229.130:41455)
0924 19:27:03.754554 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:03.754556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:03.754562 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:27:03.754579 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:03.754579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:03.754586 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:03.754630 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:03.754721 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.755125 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.755797 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.755953 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.755998 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.756099 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.756134 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.756177 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:03.756211 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:27:03.756226 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":29}
I20260924 19:27:03.756357 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.754338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41455 (local address 127.18.229.130:56297)
0924 19:27:03.754379 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.754395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:03.754448 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:03.754646 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:03.754647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:03.754648 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:27:03.754679 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.754680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.755193 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.755193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.755718 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:03.755720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:03.756011 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:03.756012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:03.756047 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:03.756051 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:27:03.756180 (+ 129us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:27:03.756180 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 19:27:03.756211 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 19:27:03.756246 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:27:03.758132 27299 raft_consensus.cc:1261] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Refusing update from remote peer 08d4271eb2ae4b66ae7ba30cfb7b7c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.758150 27019 raft_consensus.cc:1261] T 9ac9a22a4fa6403eab80331cf0f5a523 P bfad6ba329eb41659c4e2325d94919fa [term 1 FOLLOWER]: Refusing update from remote peer 08d4271eb2ae4b66ae7ba30cfb7b7c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:03.758328 27377 consensus_queue.cc:1035] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8901fed8cab4fe8aa317e8430986955" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.758446 27413 consensus_queue.cc:1035] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfad6ba329eb41659c4e2325d94919fa" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:03.768198 26860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.768287 26860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:03.768873 26860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 322a95f6e118442e98baa51f12b5d7ba: Sending DeleteTablet for 3 replicas of tablet 9ac9a22a4fa6403eab80331cf0f5a523
I20260924 19:27:03.769093 27279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac9a22a4fa6403eab80331cf0f5a523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39844
I20260924 19:27:03.769124 27138 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ac9a22a4fa6403eab80331cf0f5a523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:03 UTC) from {username='oryx', principal='oryx/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:46830
I20260924 19:27:03.769188 27393 tablet_replica.cc:330] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: stopping tablet replica
I20260924 19:27:03.769271 27393 raft_consensus.cc:2229] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:03.769316 27393 raft_consensus.cc:2258] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.769477 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 26934
I20260924 19:27:03.769649 27393 ts_tablet_manager.cc:1905] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.771318 27393 ts_tablet_manager.cc:1918] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:27:03.771384 27393 log.cc:1196] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/9ac9a22a4fa6403eab80331cf0f5a523
I20260924 19:27:03.771622 27393 ts_tablet_manager.cc:1939] T 9ac9a22a4fa6403eab80331cf0f5a523 P c8901fed8cab4fe8aa317e8430986955: Deleting consensus metadata
I20260924 19:27:03.771919 26849 catalog_manager.cc:4841] TS c8901fed8cab4fe8aa317e8430986955 (127.18.229.131:45853): tablet 9ac9a22a4fa6403eab80331cf0f5a523 (table test-table [id=624988c3e8e645c7931dab156b5fe2e8]) successfully deleted
I20260924 19:27:03.772694 27394 tablet_replica.cc:330] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: stopping tablet replica
I20260924 19:27:03.772758 27394 raft_consensus.cc:2229] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:27:03.772820 27394 raft_consensus.cc:2258] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:03.773247 27394 ts_tablet_manager.cc:1905] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:03.774636 27394 ts_tablet_manager.cc:1918] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:27:03.774693 27394 log.cc:1196] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/9ac9a22a4fa6403eab80331cf0f5a523
I20260924 19:27:03.774899 27394 ts_tablet_manager.cc:1939] T 9ac9a22a4fa6403eab80331cf0f5a523 P 08d4271eb2ae4b66ae7ba30cfb7b7c21: Deleting consensus metadata
I20260924 19:27:03.775377 26847 catalog_manager.cc:4841] TS 08d4271eb2ae4b66ae7ba30cfb7b7c21 (127.18.229.130:41455): tablet 9ac9a22a4fa6403eab80331cf0f5a523 (table test-table [id=624988c3e8e645c7931dab156b5fe2e8]) successfully deleted
W20260924 19:27:03.777856 26846 connection.cc:535] client connection to 127.18.229.129:40289 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:27:03.777968 26846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:27:03.780828 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 27074
I20260924 19:27:03.784284 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:03.778359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:40289 (local address 127.0.0.1:50646)
0924 19:27:03.784208 (+ 5849us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:03.784252 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:40289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5832,"thread_start_us":42,"threads_started":1}
W20260924 19:27:03.784402 26846 catalog_manager.cc:4584] TS bfad6ba329eb41659c4e2325d94919fa (127.18.229.129:40289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ac9a22a4fa6403eab80331cf0f5a523: Network error: Client connection negotiation failed: client connection to 127.18.229.129:40289: connect: Connection refused (error 111)
I20260924 19:27:03.787117 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 27214
I20260924 19:27:03.792747 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 26830
2026-09-24T19:27:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2798 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[27464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[27464](info): set up 2 sockets
Sep 24 19:27:03 dist-test-slave-r8cc krb5kdc[27464](info): commencing operation
krb5kdc: starting...
W20260924 19:27:04.912683 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:27:04 dist-test-slave-r8cc krb5kdc[27464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:27:05.025032 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44797
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:44797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:05.106755 27477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:05.107002 27477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:05.107041 27477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:05.108300 27477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:05.108359 27477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:05.108385 27477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:05.108417 27477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:05.108443 27477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:05.109947 27477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:44797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:05.110224 27477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:05.110461 27477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:05.115527 27482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:05.115527 27477 server_base.cc:1034] running on GCE node
W20260924 19:27:05.115542 27485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:05.115572 27483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:05.115962 27477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:05.116281 27477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:05.117429 27477 hybrid_clock.cc:648] HybridClock initialized: now 1790278025117418 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T19:27:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1328 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:27:05 dist-test-slave-r8cc krb5kdc[27490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:27:05 dist-test-slave-r8cc krb5kdc[27490](info): set up 2 sockets
Sep 24 19:27:05 dist-test-slave-r8cc krb5kdc[27490](info): commencing operation
krb5kdc: starting...
W20260924 19:27:06.144824 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:27:06.258046 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:33763
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:06.336510 27503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:06.336747 27503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:06.336783 27503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:06.337917 27503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:06.337991 27503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:06.338016 27503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:06.338037 27503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:06.338066 27503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:06.339532 27503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:06.339811 27503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:06.340034 27503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:06.344691 27509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.344760 27508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.344686 27511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:06.344887 27503 server_base.cc:1034] running on GCE node
I20260924 19:27:06.345122 27503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:06.345408 27503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:06.346555 27503 hybrid_clock.cc:648] HybridClock initialized: now 1790278026346544 us; error 35 us; skew 500 ppm
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:06.353952 27503 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:27:06.354408 27503 webserver.cc:460] Webserver started at http://127.18.229.190:38025/ using document root <none> and password file <none>
I20260924 19:27:06.354638 27503 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:06.354681 27503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:06.354821 27503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:06.355820 27503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "293dafa7a9664eef84ee016c6506aa78"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "382ad9eba760814f33f488862d9ff604"
server_key_iv: "215b980b50db2ecfd31697161c5d29b3"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.356201 27503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "293dafa7a9664eef84ee016c6506aa78"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "382ad9eba760814f33f488862d9ff604"
server_key_iv: "215b980b50db2ecfd31697161c5d29b3"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.357440 27503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:27:06.358191 27518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.358361 27503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:06.358481 27503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "293dafa7a9664eef84ee016c6506aa78"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "382ad9eba760814f33f488862d9ff604"
server_key_iv: "215b980b50db2ecfd31697161c5d29b3"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.358556 27503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:06.371116 27503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:06.373001 27503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:06.373163 27503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:06.377038 27503 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:33763
I20260924 19:27:06.377071 27570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:33763 every 8 connection(s)
I20260924 19:27:06.377413 27503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:27:06.378212 27571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:06.380437 27571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Bootstrap starting.
I20260924 19:27:06.381007 27571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:06.381300 27571 log.cc:824] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:06.381955 27571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: No bootstrap required, opened a new log
I20260924 19:27:06.382367 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 27503
I20260924 19:27:06.382455 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:27:06.382678 19350 external_mini_cluster.cc:1428] Setting key 1200f3c18d4aab6519dea2ac07b5dc2e
I20260924 19:27:06.383893 27571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:06.384059 27571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:06.384099 27571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293dafa7a9664eef84ee016c6506aa78, State: Initialized, Role: FOLLOWER
I20260924 19:27:06.384284 27571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:06.384375 27571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:27:06.384413 27571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:27:06.384461 27571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:06.385398 27571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:06.385481 27571 leader_election.cc:302] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293dafa7a9664eef84ee016c6506aa78; no voters:
I20260924 19:27:06.385679 27571 leader_election.cc:288] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:27:06.385722 27576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:06.385824 27576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 LEADER]: Becoming Leader. State: Replica: 293dafa7a9664eef84ee016c6506aa78, State: Running, Role: LEADER
I20260924 19:27:06.385901 27576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:06.385946 27571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:27:06.386214 27576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "293dafa7a9664eef84ee016c6506aa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } } }
I20260924 19:27:06.386363 27576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:06.386555 27576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 293dafa7a9664eef84ee016c6506aa78. Latest consensus state: current_term: 1 leader_uuid: "293dafa7a9664eef84ee016c6506aa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } } }
I20260924 19:27:06.386639 27576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:06.386907 27583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:06.387439 27583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:06.389034 27583 catalog_manager.cc:1347] Generated new cluster ID: 05b052f370fc46578ea1ae9dbc9f14a6
I20260924 19:27:06.389086 27583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:06.393705 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.383180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.18.229.190:33763)
0924 19:27:06.383365 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:06.383372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:06.383395 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:27:06.383873 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:06.383879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:06.383923 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:06.384030 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:06.384861 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.385938 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.387074 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.387710 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.387844 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.390370 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:06.390373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:06.390374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:06.390385 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:06.392020 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.392416 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.392418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.392419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.392535 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.392750 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.392751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.392752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.392983 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:06.393142 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:06.393249 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 19:27:06.393386 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260924 19:27:06.407042 27583 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:27:06.407544 27583 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:06.417456 27583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Generated new TSK 0
I20260924 19:27:06.417659 27583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:27:06.446535 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33763
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:06.529048 27597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:06.529245 27597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:06.529268 27597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:06.530435 27597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:06.530506 27597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:06.530562 27597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:27:06.531960 27597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:06.532214 27597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:06.532425 27597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:06.537246 27602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.537302 27603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.537250 27605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:06.537516 27597 server_base.cc:1034] running on GCE node
I20260924 19:27:06.537698 27597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:06.537880 27597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:06.539052 27597 hybrid_clock.cc:648] HybridClock initialized: now 1790278026539045 us; error 38 us; skew 500 ppm
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:06.541579 27597 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:27:06.542006 27597 webserver.cc:460] Webserver started at http://127.18.229.129:41739/ using document root <none> and password file <none>
I20260924 19:27:06.542229 27597 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:06.542299 27597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:06.542448 27597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:06.543497 27597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "6cda42118bae413b974bbba5214308ab"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "2a1926eb6c9b880016514a3a76cfcf9f"
server_key_iv: "9c058ad63e76cd4a3ff73b7d9ab9ec6b"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.543865 27597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "6cda42118bae413b974bbba5214308ab"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "2a1926eb6c9b880016514a3a76cfcf9f"
server_key_iv: "9c058ad63e76cd4a3ff73b7d9ab9ec6b"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.545238 27597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:27:06.545974 27612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.546206 27597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:06.546312 27597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "6cda42118bae413b974bbba5214308ab"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "2a1926eb6c9b880016514a3a76cfcf9f"
server_key_iv: "9c058ad63e76cd4a3ff73b7d9ab9ec6b"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.546403 27597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:06.558434 27597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:06.560258 27597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:06.560413 27597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:06.560678 27597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:06.561053 27597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:06.561090 27597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.561112 27597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:06.561125 27597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.566892 27597 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:33403
I20260924 19:27:06.566968 27725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:33403 every 8 connection(s)
I20260924 19:27:06.567278 27597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:27:06.571240 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 27597
I20260924 19:27:06.571349 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:27:06.571564 19350 external_mini_cluster.cc:1428] Setting key 00330cc146b1a22a3c7b60105ce5e5b5
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:06.577072 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.568023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:57717 (local address 127.18.229.190:33763)
0924 19:27:06.568081 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:06.568083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:06.568296 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:27:06.568824 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:06.568826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:06.568844 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:06.568891 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:06.569824 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.570330 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.571534 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.571805 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.571909 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.574161 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:06.574162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:06.574163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:06.574170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:06.575519 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.576037 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.576038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.576038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.576087 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.576431 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.576431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.576432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.576533 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:06.576585 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:06.576944 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 19:27:06.577010 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:27:06.577395 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.567997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:57717)
0924 19:27:06.568184 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:06.568217 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:06.568710 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:06.568971 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:06.568975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:06.569535 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:06.569723 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:06.569727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.570427 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.570428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:06.571430 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:06.571434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.571973 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.571974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:06.572095 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.572710 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:06.572718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:06.574048 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:06.575606 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:06.575614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:06.575616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:06.575957 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:06.576215 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:06.576216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:06.576216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:06.576363 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:06.576603 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:06.576610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:06.576798 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 19:27:06.576927 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 19:27:06.577114 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260924 19:27:06.577879 27726 heartbeater.cc:347] Connected to a master server at 127.18.229.190:33763
I20260924 19:27:06.577982 27726 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:06.578183 27726 heartbeater.cc:511] Master 127.18.229.190:33763 requested a full tablet report, sending...
I20260924 19:27:06.578691 27535 ts_manager.cc:194] Registered new tserver with Master: 6cda42118bae413b974bbba5214308ab (127.18.229.129:33403)
I20260924 19:27:06.579295 27535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:57717
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:27:06.596213 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33763
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:06.675113 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:06.675308 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:06.675330 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:06.676641 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:06.676682 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:06.676729 27731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:27:06.678171 27731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:06.678432 27731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:06.678649 27731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:06.683470 27736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.683497 27739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.683516 27737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:06.683512 27731 server_base.cc:1034] running on GCE node
I20260924 19:27:06.683847 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:06.684088 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:06.685282 27731 hybrid_clock.cc:648] HybridClock initialized: now 1790278026685270 us; error 30 us; skew 500 ppm
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:06.687877 27731 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:27:06.688359 27731 webserver.cc:460] Webserver started at http://127.18.229.130:35039/ using document root <none> and password file <none>
I20260924 19:27:06.688588 27731 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:06.688660 27731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:06.688793 27731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:06.689851 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "1b4c6e1aebf44cbdbe043eff97df3fbf"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "ce71dd41b732e35fd01d37abab40fe60"
server_key_iv: "53b77ed4804f2a7fa2ffee5e94da1d6d"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.690267 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "1b4c6e1aebf44cbdbe043eff97df3fbf"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "ce71dd41b732e35fd01d37abab40fe60"
server_key_iv: "53b77ed4804f2a7fa2ffee5e94da1d6d"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.691524 27731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:06.692251 27746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.692431 27731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:06.692543 27731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "1b4c6e1aebf44cbdbe043eff97df3fbf"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "ce71dd41b732e35fd01d37abab40fe60"
server_key_iv: "53b77ed4804f2a7fa2ffee5e94da1d6d"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.692646 27731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:06.726267 27731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:06.728195 27731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:06.728355 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:06.728583 27731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:06.728914 27731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:06.728974 27731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.729018 27731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:06.729046 27731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.734679 27731 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:33561
I20260924 19:27:06.734754 27859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:33561 every 8 connection(s)
I20260924 19:27:06.735071 27731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:27:06.740676 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 27731
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:06.740785 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:27:06.740988 19350 external_mini_cluster.cc:1428] Setting key e45bf76b9d18c975fa371d81816ad44a
I20260924 19:27:06.744302 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.735805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:51007 (local address 127.18.229.190:33763)
0924 19:27:06.735856 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:06.735859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:06.736067 (+ 208us) server_negotiation.cc:407] Connection header received
0924 19:27:06.736566 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:06.736567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:06.736577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:06.736643 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:06.737505 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.738012 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.738933 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.739108 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.739159 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.741268 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:06.741269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:06.741270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:06.741277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:06.742715 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.743184 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.743185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.743185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.743249 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.743486 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.743487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.743487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.743669 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:06.743764 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:06.744035 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 19:27:06.744185 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:27:06.744371 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.735797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:51007)
0924 19:27:06.735948 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:06.735984 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:06.736466 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:06.736697 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:06.736700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:06.737238 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:06.737433 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:06.737438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.738086 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.738086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:06.738862 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:06.738864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.739177 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.739178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:06.739270 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.739694 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:06.739701 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:06.741176 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:06.742800 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:06.742807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:06.742809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:06.743114 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:06.743322 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:06.743323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:06.743323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:06.743420 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:06.743753 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:06.743757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:06.743892 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:27:06.743985 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 19:27:06.744195 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 19:27:06.744880 27860 heartbeater.cc:347] Connected to a master server at 127.18.229.190:33763
I20260924 19:27:06.744983 27860 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:06.745157 27860 heartbeater.cc:511] Master 127.18.229.190:33763 requested a full tablet report, sending...
I20260924 19:27:06.745525 27535 ts_manager.cc:194] Registered new tserver with Master: 1b4c6e1aebf44cbdbe043eff97df3fbf (127.18.229.130:33561)
I20260924 19:27:06.745909 27535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:51007
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:27:06.764276 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33763
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:06.843318 27865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:06.843562 27865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:06.843600 27865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:06.844780 27865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:06.844854 27865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:06.844895 27865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:27:06.846272 27865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:06.846525 27865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:06.846750 27865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:06.851649 27873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:06.851683 27871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:06.851819 27865 server_base.cc:1034] running on GCE node
W20260924 19:27:06.851692 27870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:06.852043 27865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:06.852444 27865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:06.853628 27865 hybrid_clock.cc:648] HybridClock initialized: now 1790278026853611 us; error 26 us; skew 500 ppm
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:06.855970 27865 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:27:06.856418 27865 webserver.cc:460] Webserver started at http://127.18.229.131:40393/ using document root <none> and password file <none>
I20260924 19:27:06.856591 27865 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:06.856637 27865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:06.856724 27865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:06.857663 27865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "d1da95f40d0743fabf2469a759458b65"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "a370f03e1da3ee04c2d64960c3492fae"
server_key_iv: "c0a9fc4d89913d01c3ba56779ae5708f"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.857949 27865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "d1da95f40d0743fabf2469a759458b65"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "a370f03e1da3ee04c2d64960c3492fae"
server_key_iv: "c0a9fc4d89913d01c3ba56779ae5708f"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.859283 27865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:27:06.859985 27880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.860152 27865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:06.860239 27865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "d1da95f40d0743fabf2469a759458b65"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "a370f03e1da3ee04c2d64960c3492fae"
server_key_iv: "c0a9fc4d89913d01c3ba56779ae5708f"
server_key_version: "encryptionkey@0"
I20260924 19:27:06.860292 27865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:06.874758 27865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:06.876655 27865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:06.876772 27865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:06.876940 27865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:06.877235 27865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:06.877269 27865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.877290 27865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:06.877303 27865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:06.883011 27865 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:44307
I20260924 19:27:06.883081 27993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:44307 every 8 connection(s)
I20260924 19:27:06.883387 27865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:27:06.888573 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 27865
I20260924 19:27:06.888664 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:27:06.888864 19350 external_mini_cluster.cc:1428] Setting key 895ada143789c42ee8fc634ae9630584
Sep 24 19:27:06 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278026, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:06.892729 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.884088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:54173 (local address 127.18.229.190:33763)
0924 19:27:06.884172 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:06.884175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:06.884379 (+ 204us) server_negotiation.cc:407] Connection header received
0924 19:27:06.884826 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:06.884827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:06.884837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:06.884889 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:06.885827 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.886409 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.887596 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.887861 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.887928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.890177 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:06.890179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:06.890179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:06.890186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:06.891325 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.891729 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.891730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.891731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.891793 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:06.892039 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:06.892040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:06.892040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:06.892215 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:06.892289 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:06.892559 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:27:06.892644 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:27:06.892941 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:06.884080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:54173)
0924 19:27:06.884263 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:06.884297 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:06.884750 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:06.884939 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:06.884942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:06.885563 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:06.885755 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:06.885760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.886552 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.886553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:06.887484 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:06.887487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:06.887943 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:06.887944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:06.888037 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:06.888579 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:06.888587 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:06.890065 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:06.891421 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:06.891427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:06.891429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:06.891646 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:06.891867 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:06.891868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:06.891868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:06.891969 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:06.892280 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:06.892285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:06.892419 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:27:06.892538 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 19:27:06.892684 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260924 19:27:06.893380 27994 heartbeater.cc:347] Connected to a master server at 127.18.229.190:33763
I20260924 19:27:06.893467 27994 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:06.893592 27994 heartbeater.cc:511] Master 127.18.229.190:33763 requested a full tablet report, sending...
I20260924 19:27:06.893925 27535 ts_manager.cc:194] Registered new tserver with Master: d1da95f40d0743fabf2469a759458b65 (127.18.229.131:44307)
I20260924 19:27:06.894299 27535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:54173
I20260924 19:27:06.900511 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab.
Entry for principal oryx/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab.
WARNING: no policy specified for oryx/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab.
Entry for principal oryx/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab.
WARNING: no policy specified for oryx/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab.
Entry for principal oryx/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab.
I20260924 19:27:06.963573 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 27597
I20260924 19:27:06.967944 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 27731
I20260924 19:27:06.972287 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 27865
I20260924 19:27:06.976258 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 27503
I20260924 19:27:06.981298 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:33763
--webserver_interface=127.18.229.190
--webserver_port=38025
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:07.059456 28003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:07.059649 28003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:07.059671 28003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:07.060760 28003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:07.060793 28003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:07.060807 28003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:07.060820 28003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:07.060832 28003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:07.062371 28003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=38025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:07.062559 28003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:07.062731 28003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:07.067346 28008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:07.067354 28009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.067406 28003 server_base.cc:1034] running on GCE node
W20260924 19:27:07.067354 28011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.067842 28003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:07.068190 28003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:07.069365 28003 hybrid_clock.cc:648] HybridClock initialized: now 1790278027069349 us; error 28 us; skew 500 ppm
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278027, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:07.071758 28003 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:27:07.072270 28003 webserver.cc:460] Webserver started at http://127.18.229.190:38025/ using document root <none> and password file <none>
I20260924 19:27:07.072530 28003 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:07.072611 28003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:07.073930 28003 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:27:07.074579 28018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.074759 28003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:07.074847 28003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "293dafa7a9664eef84ee016c6506aa78"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "382ad9eba760814f33f488862d9ff604"
server_key_iv: "215b980b50db2ecfd31697161c5d29b3"
server_key_version: "encryptionkey@0"
I20260924 19:27:07.075201 28003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:07.099926 28003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:07.101797 28003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:07.101961 28003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:07.105971 28003 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:33763
I20260924 19:27:07.106045 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:33763 every 8 connection(s)
I20260924 19:27:07.106431 28003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:27:07.107815 28071 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:27:07.108453 28071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Bootstrap starting.
I20260924 19:27:07.116133 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28003
I20260924 19:27:07.116477 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:33403
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:33763
--webserver_port=41739
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:27:07.116653 28071 log.cc:824] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:07.118491 28071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:27:07.118779 28071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Bootstrap complete.
I20260924 19:27:07.121135 28071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:07.121366 28071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293dafa7a9664eef84ee016c6506aa78, State: Initialized, Role: FOLLOWER
I20260924 19:27:07.121531 28071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:07.121599 28071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:27:07.121659 28071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:27:07.121701 28071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:27:07.122700 28071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:07.122786 28071 leader_election.cc:302] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293dafa7a9664eef84ee016c6506aa78; no voters:
I20260924 19:27:07.122968 28071 leader_election.cc:288] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:27:07.123005 28075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:27:07.123113 28075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [term 2 LEADER]: Becoming Leader. State: Replica: 293dafa7a9664eef84ee016c6506aa78, State: Running, Role: LEADER
I20260924 19:27:07.123200 28075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } }
I20260924 19:27:07.123245 28071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:27:07.123476 28077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 293dafa7a9664eef84ee016c6506aa78. Latest consensus state: current_term: 2 leader_uuid: "293dafa7a9664eef84ee016c6506aa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } } }
I20260924 19:27:07.123737 28077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:07.123660 28076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "293dafa7a9664eef84ee016c6506aa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293dafa7a9664eef84ee016c6506aa78" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 33763 } } }
I20260924 19:27:07.124014 28076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:07.124086 28085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:07.124642 28085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:07.124797 28085 catalog_manager.cc:1259] Loaded cluster ID: 05b052f370fc46578ea1ae9dbc9f14a6
I20260924 19:27:07.124840 28085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:27:07.125762 28085 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:07.125946 28085 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 293dafa7a9664eef84ee016c6506aa78: Loaded TSK: 0
I20260924 19:27:07.126156 28085 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:27:07.216943 28074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:07.217180 28074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:07.217217 28074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:07.218381 28074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:07.218452 28074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:07.218498 28074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:27:07.219928 28074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.129.keytab
--principal=oryx/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:33403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=41739
--tserver_master_addrs=127.18.229.190:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:07.220202 28074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:07.220429 28074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:07.226284 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:07.226322 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:07.226604 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.226667 28074 server_base.cc:1034] running on GCE node
I20260924 19:27:07.226902 28074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:07.227178 28074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:07.228332 28074 hybrid_clock.cc:648] HybridClock initialized: now 1790278027228320 us; error 31 us; skew 500 ppm
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278027, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:07.230792 28074 init.cc:377] Logged in from keytab as oryx/127.18.229.129@KRBTEST.COM (short username oryx)
I20260924 19:27:07.231285 28074 webserver.cc:460] Webserver started at http://127.18.229.129:41739/ using document root <none> and password file <none>
I20260924 19:27:07.231515 28074 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:07.231594 28074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:07.232993 28074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:07.233623 28107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.233814 28074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:07.233909 28074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "6cda42118bae413b974bbba5214308ab"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "2a1926eb6c9b880016514a3a76cfcf9f"
server_key_iv: "9c058ad63e76cd4a3ff73b7d9ab9ec6b"
server_key_version: "encryptionkey@0"
I20260924 19:27:07.234249 28074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:07.257277 28074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:07.259459 28074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:07.259624 28074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:07.259893 28074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:07.260284 28074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:07.260339 28074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.260381 28074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:07.260407 28074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.267805 28074 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:33403
I20260924 19:27:07.267877 28220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:33403 every 8 connection(s)
I20260924 19:27:07.268296 28074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:27:07.271845 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28074
I20260924 19:27:07.272213 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:33561
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:33763
--webserver_port=35039
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.277889 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.269067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:60771)
0924 19:27:07.269245 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.269291 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.269758 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.270205 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.270208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.270773 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.270989 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.270994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.272150 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.272151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.273514 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.273518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.274382 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.274383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.274480 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.274995 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.275005 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.277668 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
W20260924 19:27:07.277943 28224 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.269119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:60771 (local address 127.18.229.190:33763)
0924 19:27:07.269336 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.269344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.269385 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:27:07.269962 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.269969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.270027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.270170 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.271100 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.271997 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.273617 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.274287 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.274416 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.277791 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:60771: BlockingRecv error: recv got EOF from 127.18.229.129:60771 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
W20260924 19:27:07.278162 28221 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:33763 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.229.190:33763: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.282965 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.278373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:35205)
0924 19:27:07.278427 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.278439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.278482 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.278710 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.278710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.278845 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.278890 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.278891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.279621 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.279622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.280221 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.280223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.280616 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.280617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.280684 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.281180 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.281185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.282895 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 19:27:07.283133 28224 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.278440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:35205 (local address 127.18.229.190:33763)
0924 19:27:07.278588 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.278590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.278597 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:27:07.278618 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.278619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.278632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.278706 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.278978 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.279508 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.280314 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.280527 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.280615 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.282978 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:35205: BlockingRecv error: recv got EOF from 127.18.229.129:35205 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.287724 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.283316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:33943)
0924 19:27:07.283378 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.283388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.283422 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.283559 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.283560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.283678 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.283704 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.283705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.284629 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.284630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.285092 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.285093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.285550 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.285551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.285593 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.285999 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.286003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.287659 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:27:07.287936 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.283369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:33943 (local address 127.18.229.190:33763)
0924 19:27:07.283436 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.283438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.283446 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:07.283468 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.283469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.283480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.283556 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.283769 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.284561 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.285163 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.285485 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.285546 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.287877 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:33943: BlockingRecv error: recv got EOF from 127.18.229.129:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 19:27:07.287992 28221 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:27:07.355028 28225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:07.355263 28225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:07.355300 28225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:07.356467 28225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:07.356536 28225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:07.356581 28225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:27:07.358039 28225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.130.keytab
--principal=oryx/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:33561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=35039
--tserver_master_addrs=127.18.229.190:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:07.358310 28225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:07.358532 28225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:07.363534 28231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:07.363567 28232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.363585 28225 server_base.cc:1034] running on GCE node
W20260924 19:27:07.363543 28234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.363915 28225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:07.364189 28225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:07.365345 28225 hybrid_clock.cc:648] HybridClock initialized: now 1790278027365329 us; error 38 us; skew 500 ppm
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278027, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:07.367717 28225 init.cc:377] Logged in from keytab as oryx/127.18.229.130@KRBTEST.COM (short username oryx)
I20260924 19:27:07.368178 28225 webserver.cc:460] Webserver started at http://127.18.229.130:35039/ using document root <none> and password file <none>
I20260924 19:27:07.368399 28225 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:07.368477 28225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:07.369774 28225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:27:07.370303 28241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.370460 28225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.370554 28225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "1b4c6e1aebf44cbdbe043eff97df3fbf"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "ce71dd41b732e35fd01d37abab40fe60"
server_key_iv: "53b77ed4804f2a7fa2ffee5e94da1d6d"
server_key_version: "encryptionkey@0"
I20260924 19:27:07.370913 28225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:07.384806 28225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:07.386571 28225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:07.386737 28225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:07.386962 28225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:07.387267 28225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:07.387320 28225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.387353 28225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:07.387380 28225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.392983 28225 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:33561
I20260924 19:27:07.393030 28354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:33561 every 8 connection(s)
I20260924 19:27:07.393356 28225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:27:07.397675 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28225
I20260924 19:27:07.397993 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:44307
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:33763
--webserver_port=40393
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:27:07.400756 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.394155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:52795 (local address 127.18.229.190:33763)
0924 19:27:07.394214 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.394216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.394445 (+ 229us) server_negotiation.cc:407] Connection header received
0924 19:27:07.394918 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.394919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.394930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.394983 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.395839 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.396361 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.397286 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.397485 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.397554 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.400703 (+ 3149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:52795: BlockingRecv error: recv got EOF from 127.18.229.130:52795 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:27:07.400811 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.394127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:52795)
0924 19:27:07.394318 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.394358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.394838 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.395036 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.395039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.395580 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.395765 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.395770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.396436 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.396437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.397215 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.397217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.397562 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.397563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.397657 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.398225 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.398235 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.400621 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
W20260924 19:27:07.401044 28355 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:33763 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.229.190:33763: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.405788 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.401244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:35313)
0924 19:27:07.401294 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.401305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.401344 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.401477 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.401478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.401585 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.401626 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.401627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.402596 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.402597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.403067 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.403068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.403565 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.403566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.403610 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.404018 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.404023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.405746 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:27:07.405962 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.401314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:35313 (local address 127.18.229.190:33763)
0924 19:27:07.401356 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.401358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.401368 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:27:07.401397 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.401398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.401408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.401474 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.401697 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.402482 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.403138 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.403493 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.403572 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.405867 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:35313: BlockingRecv error: recv got EOF from 127.18.229.130:35313 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.410509 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.406223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:53231)
0924 19:27:07.406321 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.406331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.406370 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.406500 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.406501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.406598 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.406627 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.406628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.407520 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.407521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.407967 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.407969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.408518 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.408519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.408553 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.408953 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.408956 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.410471 (+ 1515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 19:27:07.410657 28355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:27:07.410689 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.406284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:53231 (local address 127.18.229.190:33763)
0924 19:27:07.406346 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.406348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.406361 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:27:07.406426 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.406427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.406439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.406494 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.406697 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.407446 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.408045 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.408444 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.408524 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.410577 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:53231: BlockingRecv error: recv got EOF from 127.18.229.130:53231 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 19:27:07.481369 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:07.481570 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:07.481594 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:07.482762 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:07.482805 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:07.482859 28358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:27:07.484321 28358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/krb5kdc/oryx_127.18.229.131.keytab
--principal=oryx/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:44307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=40393
--tserver_master_addrs=127.18.229.190:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:07.484552 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:07.484778 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:07.489763 28363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:07.489750 28366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.489933 28358 server_base.cc:1034] running on GCE node
W20260924 19:27:07.489768 28364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:07.490226 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:07.490429 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:07.491575 28358 hybrid_clock.cc:648] HybridClock initialized: now 1790278027491563 us; error 27 us; skew 500 ppm
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278027, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:07.494095 28358 init.cc:377] Logged in from keytab as oryx/127.18.229.131@KRBTEST.COM (short username oryx)
I20260924 19:27:07.494568 28358 webserver.cc:460] Webserver started at http://127.18.229.131:40393/ using document root <none> and password file <none>
I20260924 19:27:07.494794 28358 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:07.494875 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:07.496213 28358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:07.496783 28373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.496943 28358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:07.497045 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "d1da95f40d0743fabf2469a759458b65"
format_stamp: "Formatted at 2026-09-24 19:27:06 on dist-test-slave-r8cc"
server_key: "a370f03e1da3ee04c2d64960c3492fae"
server_key_iv: "c0a9fc4d89913d01c3ba56779ae5708f"
server_key_version: "encryptionkey@0"
I20260924 19:27:07.497392 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:07.508598 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:07.510795 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:07.510946 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:07.511164 28358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:07.511471 28358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:07.511526 28358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.511565 28358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:07.511592 28358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:07.517488 28358 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:44307
I20260924 19:27:07.517539 28486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:44307 every 8 connection(s)
I20260924 19:27:07.517855 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:27:07.522919 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28358
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:27:07.525235 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.518507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:43363 (local address 127.18.229.190:33763)
0924 19:27:07.518563 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.518565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.518794 (+ 229us) server_negotiation.cc:407] Connection header received
0924 19:27:07.519216 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.519217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.519226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.519285 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.520212 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.520690 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.521636 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.521826 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.521887 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.525189 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:43363: BlockingRecv error: recv got EOF from 127.18.229.131:43363 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:27:07.525305 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.518467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:43363)
0924 19:27:07.518632 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.518672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.519131 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.519361 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.519365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.519901 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.520085 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.520089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.520764 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.520764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.521524 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.521526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.521902 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.521902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.521998 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.522424 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.522431 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.525117 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
W20260924 19:27:07.525511 28487 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:33763 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.229.190:33763: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.530021 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.525705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:33803)
0924 19:27:07.525810 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.525819 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.525864 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.525979 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.525980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.526121 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.526166 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.526167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.526837 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.526837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.527565 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.527568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.527887 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.527888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.527959 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.528381 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.528385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.529979 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 19:27:07.530148 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.525766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:33803 (local address 127.18.229.190:33763)
0924 19:27:07.525821 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.525822 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.525852 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:27:07.525918 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.525918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.525925 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.525966 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.526228 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.526764 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.527646 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.527815 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.527866 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.530081 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:33803: BlockingRecv error: recv got EOF from 127.18.229.131:33803 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:27:07.531644 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.523426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.18.229.190:33763)
0924 19:27:07.523479 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.523481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.523599 (+ 118us) server_negotiation.cc:407] Connection header received
0924 19:27:07.523683 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.523684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.523698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.523755 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.524546 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.525249 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.526589 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.526881 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.526939 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.528306 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:07.528310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:07.528311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:07.528323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:07.530248 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:07.530645 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:07.530648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:07.530648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:07.530741 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:07.530946 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:07.530948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:07.530948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:07.531173 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:07.531328 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:07.531438 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:27:07.531543 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:07 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:07.534354 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:07.530348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:56617)
0924 19:27:07.530385 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:07.530396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:07.530432 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:07.530542 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:07.530542 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:07.530664 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:07.530687 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.530688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.531287 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.531288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.531943 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:07.531946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.532397 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.532398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:07.532450 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.532860 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:07.532865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:07.534315 (+ 1450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 19:27:07.534477 28226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:07.530410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:56617 (local address 127.18.229.190:33763)
0924 19:27:07.530442 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:07.530443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:07.530451 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:07.530476 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:07.530477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:07.530493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:07.530540 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:07.530758 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.531194 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.532022 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:07.532321 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:07.532395 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:07.534407 (+ 2012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:56617: BlockingRecv error: recv got EOF from 127.18.229.131:56617 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:27:07.534473 28487 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 19:27:08 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:08 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:08.292975 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:08.288364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:44977)
0924 19:27:08.288503 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:08.288513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:08.288569 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:08.288699 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:08.288700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:08.288853 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:08.288900 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:08.288902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.289562 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.289562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:08.290364 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:08.290367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.290689 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.290690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:08.290728 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:08.291142 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:08.291146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:08.292906 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
W20260924 19:27:08.293076 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:08.288459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:44977 (local address 127.18.229.190:33763)
0924 19:27:08.288581 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:08.288583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:08.288591 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:08.288618 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:08.288618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:08.288628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:08.288685 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:08.288961 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.289458 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.290428 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.290627 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.290681 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:08.293000 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:44977: BlockingRecv error: recv got EOF from 127.18.229.129:44977 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 24 19:27:08 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:08 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:08.415431 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:08.411038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:45403)
0924 19:27:08.411206 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:08.411217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:08.411279 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:08.411408 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:08.411408 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:08.411577 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:08.411625 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:08.411627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.412269 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.412270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:08.412831 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:08.412833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.413138 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.413139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:08.413179 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:08.413678 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:08.413683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:08.415366 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
W20260924 19:27:08.415544 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:08.411105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:45403 (local address 127.18.229.190:33763)
0924 19:27:08.411182 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:08.411184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:08.411276 (+ 92us) server_negotiation.cc:407] Connection header received
0924 19:27:08.411326 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:08.411327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:08.411337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:08.411386 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:08.411682 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.412203 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.412897 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.413076 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.413131 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:08.415447 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:45403: BlockingRecv error: recv got EOF from 127.18.229.130:45403 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 19:27:08 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:08 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:08.539660 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:08.534945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:55965)
0924 19:27:08.535112 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:08.535131 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:08.535185 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:08.535316 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:08.535316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:08.535466 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:08.535512 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:08.535514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.536096 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.536097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:08.536779 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:08.536782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.537081 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.537081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:08.537126 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:08.537613 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:08.537617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:08.539548 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
W20260924 19:27:08.539692 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:08.535135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:55965 (local address 127.18.229.190:33763)
0924 19:27:08.535194 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:08.535196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:08.535205 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:08.535235 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:08.535236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:08.535247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:08.535295 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:08.535570 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.536037 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.536849 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:08.537026 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:08.537073 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:08.539627 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:55965: BlockingRecv error: recv got EOF from 127.18.229.131:55965 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:27:09 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:09 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:09.297652 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:09.293426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:34737)
0924 19:27:09.293546 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:09.293560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:09.293639 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:09.293764 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:09.293765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:09.293916 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:09.293963 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:09.293964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.294575 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.294576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:09.295091 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:09.295092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.295400 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.295400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:09.295437 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:09.295872 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:09.295876 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:09.297586 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
W20260924 19:27:09.297839 28496 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:09.293504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:34737 (local address 127.18.229.190:33763)
0924 19:27:09.293652 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:09.293654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:09.293662 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:09.293682 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:09.293683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:09.293693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:09.293748 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:09.294026 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.294513 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.295157 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.295340 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.295388 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:09.297720 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:34737: BlockingRecv error: recv got EOF from 127.18.229.129:34737 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
Sep 24 19:27:09 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:09 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:09.423533 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:09.415839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:57379)
0924 19:27:09.415963 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:09.415974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:09.416037 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:09.416224 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:09.416225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:09.416374 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:09.416423 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:09.416424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.417059 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.417060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:09.417731 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:09.417735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.418165 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.418166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:09.418208 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:09.421751 (+ 3543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:09.421761 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:09.423478 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
W20260924 19:27:09.423766 28496 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:09.415922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57379 (local address 127.18.229.190:33763)
0924 19:27:09.415973 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:09.415974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:09.416024 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:27:09.416092 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:09.416092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:09.416139 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:09.416191 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:09.416483 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.416996 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.417806 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.418099 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.418157 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:09.423623 (+ 5466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:57379: BlockingRecv error: recv got EOF from 127.18.229.130:57379 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 19:27:09 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:09 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:09.544385 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:09.540265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:53071)
0924 19:27:09.540404 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:09.540422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:09.540478 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:09.540599 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:09.540600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:09.540751 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:09.540797 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:09.540798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.541421 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.541421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:09.541919 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:09.541920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.542219 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.542220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:09.542259 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:09.542657 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:09.542661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:09.544328 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
W20260924 19:27:09.544492 28496 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:09.540363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:53071 (local address 127.18.229.190:33763)
0924 19:27:09.540434 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:09.540437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:09.540464 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:27:09.540534 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:09.540535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:09.540546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:09.540588 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:09.540863 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.541351 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.541984 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:09.542158 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:09.542206 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:09.544417 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:53071: BlockingRecv error: recv got EOF from 127.18.229.131:53071 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:27:10 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:10 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:10.302872 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:10.298242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:53421)
0924 19:27:10.298376 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:10.298394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:10.298439 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:10.298575 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:10.298575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:10.298728 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:10.298777 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:10.298779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.299421 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.299421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:10.299994 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:10.299996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.300433 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.300434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:10.300479 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:10.301037 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:10.301043 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:10.302814 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
W20260924 19:27:10.303036 28500 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:10.298334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:53421 (local address 127.18.229.190:33763)
0924 19:27:10.298458 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:10.298460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:10.298467 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:27:10.298488 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:10.298489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:10.298498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:10.298559 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:10.298840 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.299354 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.300085 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.300366 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.300420 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:10.302934 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:53421: BlockingRecv error: recv got EOF from 127.18.229.129:53421 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Sep 24 19:27:10 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:10 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:10.428668 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:10.423967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:40451)
0924 19:27:10.424182 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:10.424200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:10.424279 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:10.424485 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:10.424486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:10.424728 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:10.424786 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:10.424788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.425456 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.425456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:10.425964 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:10.425965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.426335 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.426336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:10.426375 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:10.426794 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:10.426798 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:10.428611 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":110,"threads_started":1}
W20260924 19:27:10.428779 28500 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:10.424083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:40451 (local address 127.18.229.190:33763)
0924 19:27:10.424145 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:10.424147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:10.424256 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:27:10.424338 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:10.424339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:10.424352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:10.424439 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:10.424885 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.425386 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.426063 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.426262 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.426330 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:10.428705 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:40451: BlockingRecv error: recv got EOF from 127.18.229.130:40451 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 19:27:10 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:10 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:10.549679 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:10.544918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:34237)
0924 19:27:10.545103 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:10.545120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:10.545198 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:10.545348 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:10.545349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:10.545521 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:10.545565 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:10.545567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.546224 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.546225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:10.546786 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:10.546787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.547175 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.547175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:10.547239 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:10.547850 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:10.547855 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:10.549588 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
W20260924 19:27:10.549741 28500 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:10.545012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:34237 (local address 127.18.229.190:33763)
0924 19:27:10.545084 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:10.545086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:10.545181 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:27:10.545276 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:10.545277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:10.545291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:10.545344 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:10.545644 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.546136 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.546887 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:10.547088 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:10.547173 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:10.549678 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:34237: BlockingRecv error: recv got EOF from 127.18.229.131:34237 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 19:27:11 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:11 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:11.307647 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:11.303393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:48881)
0924 19:27:11.303530 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:11.303544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:11.303606 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:11.303729 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:11.303730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:11.303881 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:11.303928 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:11.303930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.304601 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.304602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:11.305146 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:11.305147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.305449 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.305449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:11.305487 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:11.305898 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:11.305902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:11.307591 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
W20260924 19:27:11.307801 28504 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:11.303433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:48881 (local address 127.18.229.190:33763)
0924 19:27:11.303599 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:11.303601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:11.303609 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:11.303648 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:11.303649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:11.303657 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:11.303713 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:11.304003 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.304537 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.305213 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.305389 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.305436 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:11.307719 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:48881: BlockingRecv error: recv got EOF from 127.18.229.129:48881 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
Sep 24 19:27:11 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:11 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:11.438915 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:11.429130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:50955)
0924 19:27:11.429288 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:11.429305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:11.429391 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:11.429537 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:11.429538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:11.429705 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:11.429751 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:11.429753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.430423 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.430424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:11.430951 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:11.430953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.431340 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.431341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:11.431407 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:11.437101 (+ 5694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:11.437106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:11.438822 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
W20260924 19:27:11.438968 28504 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:11.429205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:50955 (local address 127.18.229.190:33763)
0924 19:27:11.429274 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:11.429277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:11.429375 (+ 98us) server_negotiation.cc:407] Connection header received
0924 19:27:11.429460 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:11.429461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:11.429473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:11.429530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:11.429831 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.430343 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.431038 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.431237 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.431326 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:11.438906 (+ 7580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:50955: BlockingRecv error: recv got EOF from 127.18.229.130:50955 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 19:27:11 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:11 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:11.554864 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:11.550238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:40871)
0924 19:27:11.550349 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:11.550360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:11.550438 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:11.550639 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:11.550640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:11.550844 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:11.550904 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:11.550906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.551519 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.551520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:11.552030 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:11.552031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.552352 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.552352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:11.552391 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:11.552970 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:11.552974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:11.554797 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
W20260924 19:27:11.554932 28504 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:11.550300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:40871 (local address 127.18.229.190:33763)
0924 19:27:11.550354 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:11.550356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:11.550438 (+ 82us) server_negotiation.cc:407] Connection header received
0924 19:27:11.550534 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:11.550535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:11.550548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:11.550617 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:11.550971 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.551432 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.552099 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:11.552294 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:11.552343 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:11.554885 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:40871: BlockingRecv error: recv got EOF from 127.18.229.131:40871 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 19:27:12 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:12 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:12.312732 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:12.308089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:42847)
0924 19:27:12.308269 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:12.308280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:12.308343 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:12.308484 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:12.308485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:12.308639 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:12.308687 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:12.308688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.309304 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.309305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:12.310069 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:12.310072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.310402 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.310402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:12.310441 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:12.310918 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:12.310923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:12.312676 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
W20260924 19:27:12.312883 28508 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:12.308218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:42847 (local address 127.18.229.190:33763)
0924 19:27:12.308315 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:12.308316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:12.308324 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:12.308388 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:12.308388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:12.308397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:12.308462 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:12.308749 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.309230 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.310146 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.310341 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.310393 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:12.312797 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:42847: BlockingRecv error: recv got EOF from 127.18.229.129:42847 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
Sep 24 19:27:12 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:12 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:12.444046 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:12.439404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:44393)
0924 19:27:12.439537 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:12.439548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:12.439600 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:12.439795 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:12.439796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:12.439970 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:12.440017 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:12.440018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.440741 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.440742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:12.441261 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:12.441263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.441627 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.441629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:12.441713 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:12.442154 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:12.442159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:12.443947 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260924 19:27:12.444092 28508 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:12.439551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:44393 (local address 127.18.229.190:33763)
0924 19:27:12.439617 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:12.439619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:12.439630 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:27:12.439665 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:12.439666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:12.439676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:12.439734 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:12.440152 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.440655 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.441347 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.441540 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.441620 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:12.444029 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:44393: BlockingRecv error: recv got EOF from 127.18.229.130:44393 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:27:12 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:12 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:12.560060 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:12.555394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:34735)
0924 19:27:12.555520 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:12.555532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:12.555607 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:12.555773 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:12.555774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:12.555978 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:12.556059 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:12.556061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.556767 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.556769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:12.557320 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:12.557322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.557712 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.557713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:12.557771 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:12.558195 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:12.558199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:12.560002 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260924 19:27:12.560195 28508 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:12.555449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:34735 (local address 127.18.229.190:33763)
0924 19:27:12.555502 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:12.555504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:12.555614 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:27:12.555686 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:12.555687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:12.555701 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:12.555763 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:12.556178 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.556683 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.557410 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:12.557626 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:12.557712 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:12.560090 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:34735: BlockingRecv error: recv got EOF from 127.18.229.131:34735 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 19:27:13 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:13 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:13.318307 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:13.313166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:59631)
0924 19:27:13.313355 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:13.313368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:13.313439 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:13.313637 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:13.313638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:13.313854 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:13.313911 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:13.313915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.314740 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.314741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:13.315543 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:13.315546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.315865 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.315866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:13.315905 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:13.316445 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:13.316450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:13.318212 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":56,"threads_started":1}
W20260924 19:27:13.318358 28512 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:13.313231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:59631 (local address 127.18.229.190:33763)
0924 19:27:13.313358 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:13.313361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:13.313463 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:27:13.313493 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:13.313494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:13.313507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:13.313575 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:13.314000 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.314656 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.315605 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.315800 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.315856 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:13.318288 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:59631: BlockingRecv error: recv got EOF from 127.18.229.129:59631 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
Sep 24 19:27:13 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:13 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:13.449262 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:13.444663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:50761)
0924 19:27:13.444832 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:13.444843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:13.444907 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:13.445068 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:13.445068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:13.445257 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:13.445302 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:13.445304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.445970 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.445971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:13.446499 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:13.446502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.446883 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.446883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:13.446942 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:13.447433 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:13.447439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:13.449166 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
W20260924 19:27:13.449319 28512 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:13.444742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:50761 (local address 127.18.229.190:33763)
0924 19:27:13.444810 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:13.444813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:13.444887 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:27:13.444972 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:13.444973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:13.444985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:13.445062 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:13.445380 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.445883 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.446582 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.446795 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.446881 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:13.449256 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:50761: BlockingRecv error: recv got EOF from 127.18.229.130:50761 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 19:27:13 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:13 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:13.565388 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:13.560686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:53307)
0924 19:27:13.560892 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:13.560904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:13.560980 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:13.561142 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:13.561143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:13.561324 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:13.561370 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:13.561371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.562056 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.562058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:13.562584 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:13.562585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.562958 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.562959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:13.563018 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:13.563530 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:13.563535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:13.565294 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
W20260924 19:27:13.565443 28512 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:13.560785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:53307 (local address 127.18.229.190:33763)
0924 19:27:13.560884 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:13.560886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:13.560960 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:27:13.561037 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:13.561038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:13.561051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:13.561135 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:13.561451 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.561963 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.562673 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:13.562873 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:13.562958 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:13.565380 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:53307: BlockingRecv error: recv got EOF from 127.18.229.131:53307 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 19:27:14 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:14 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:14.323406 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:14.318863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:52179)
0924 19:27:14.318998 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:14.319012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:14.319067 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:14.319209 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:14.319210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:14.319362 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:14.319409 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:14.319410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.320029 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.320030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:14.320556 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:14.320558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.320874 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.320875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:14.320913 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:14.321350 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:14.321355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:14.323278 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
W20260924 19:27:14.323453 28516 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:14.318965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:52179 (local address 127.18.229.190:33763)
0924 19:27:14.319087 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:14.319093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:14.319101 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:14.319123 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:14.319124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:14.319133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:14.319195 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:14.319475 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.319968 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.320628 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.320813 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.320862 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:14.323385 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:52179: BlockingRecv error: recv got EOF from 127.18.229.129:52179 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 24 19:27:14 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:14 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:14.454782 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:14.449873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:45627)
0924 19:27:14.450042 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:14.450053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:14.450133 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:14.450271 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:14.450272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:14.450429 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:14.450477 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:14.450479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.451156 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.451157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:14.451664 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:14.451666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.452000 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.452001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:14.452050 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:14.452795 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:14.452800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:14.454725 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
W20260924 19:27:14.454919 28516 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:14.449937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:45627 (local address 127.18.229.190:33763)
0924 19:27:14.450054 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:14.450056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:14.450119 (+ 63us) server_negotiation.cc:407] Connection header received
0924 19:27:14.450195 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:14.450196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:14.450207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:14.450248 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:14.450578 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.451069 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.451736 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.451917 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.451970 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:14.454838 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:45627: BlockingRecv error: recv got EOF from 127.18.229.130:45627 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 19:27:14 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:14 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:14.570456 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:14.565899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:43743)
0924 19:27:14.566106 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:14.566116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:14.566204 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:14.566326 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:14.566327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:14.566517 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:14.566562 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:14.566564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.567224 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.567225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:14.567761 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:14.567762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.568154 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.568155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:14.568220 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:14.568661 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:14.568665 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:14.570363 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
W20260924 19:27:14.570505 28516 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:14.566026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:43743 (local address 127.18.229.190:33763)
0924 19:27:14.566099 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:14.566102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:14.566187 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:27:14.566258 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:14.566259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:14.566273 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:14.566323 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:14.566643 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.567133 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.567845 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:14.568045 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:14.568154 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:14.570441 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:43743: BlockingRecv error: recv got EOF from 127.18.229.131:43743 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 19:27:15 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:15 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:15.328433 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:15.323905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:46773)
0924 19:27:15.324047 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:15.324057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:15.324127 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:15.324336 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:15.324336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:15.324502 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:15.324551 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:15.324553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.325193 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.325193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:15.325718 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:15.325721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.326047 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.326048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:15.326086 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:15.326586 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:15.326590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:15.328369 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260924 19:27:15.328625 28520 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:15.324015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:46773 (local address 127.18.229.190:33763)
0924 19:27:15.324171 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:15.324172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:15.324179 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:27:15.324199 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:15.324200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:15.324209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:15.324312 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:15.324621 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.325126 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.325791 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.325982 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.326035 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:15.328454 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:46773: BlockingRecv error: recv got EOF from 127.18.229.129:46773 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
Sep 24 19:27:15 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:15 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:15.460561 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:15.455442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:40371)
0924 19:27:15.455634 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:15.455646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:15.455746 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:15.455928 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:15.455929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:15.456152 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:15.456211 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:15.456218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.457026 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.457028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:15.457570 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:15.457571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.457968 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.457969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:15.458028 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:15.458528 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:15.458534 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:15.460446 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
W20260924 19:27:15.460661 28520 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:15.455530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:40371 (local address 127.18.229.190:33763)
0924 19:27:15.455627 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:15.455630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:15.455726 (+ 96us) server_negotiation.cc:407] Connection header received
0924 19:27:15.455834 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:15.455835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:15.455847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:15.455916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:15.456296 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.456937 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.457657 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.457860 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.457956 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:15.460590 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:40371: BlockingRecv error: recv got EOF from 127.18.229.130:40371 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 19:27:15 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:15 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:15.575364 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:15.570929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:56823)
0924 19:27:15.571079 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:15.571089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:15.571147 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:15.571266 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:15.571267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:15.571416 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:15.571463 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:15.571465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.572073 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.572074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:15.572659 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:15.572661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.572969 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.572970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:15.573006 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:15.573516 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:15.573521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:15.575299 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260924 19:27:15.575460 28520 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:15.570972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:56823 (local address 127.18.229.190:33763)
0924 19:27:15.571019 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:15.571022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:15.571150 (+ 128us) server_negotiation.cc:407] Connection header received
0924 19:27:15.571188 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:15.571188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:15.571201 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:15.571247 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:15.571522 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.572002 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.572727 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:15.572907 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:15.572958 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:15.575392 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:56823: BlockingRecv error: recv got EOF from 127.18.229.131:56823 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 19:27:16 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:16 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:16.333390 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:16.329010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:33969)
0924 19:27:16.329142 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:16.329153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:16.329210 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:16.329357 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:16.329358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:16.329518 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:16.329564 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:16.329566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.330165 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.330166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:16.330670 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:16.330671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.330967 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.330967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:16.331015 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:16.331514 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:16.331520 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:16.333333 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
W20260924 19:27:16.333526 28524 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:16.329071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:33969 (local address 127.18.229.190:33763)
0924 19:27:16.329189 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:16.329190 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:16.329200 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:27:16.329265 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:16.329265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:16.329274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:16.329341 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:16.329624 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.330102 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.330732 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.330907 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.330953 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:16.333415 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:33969: BlockingRecv error: recv got EOF from 127.18.229.129:33969 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
Sep 24 19:27:16 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:16 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:16.471300 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:16.461232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:43645)
0924 19:27:16.461399 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:16.461412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:16.461487 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:16.461628 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:16.461629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:16.461785 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:16.461838 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:16.461842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.462455 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.462456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:16.462957 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:16.462958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.463271 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.463272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:16.463310 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:16.469350 (+ 6040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:16.469356 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:16.471242 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
W20260924 19:27:16.471410 28524 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:16.461434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:43645 (local address 127.18.229.190:33763)
0924 19:27:16.461500 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:16.461502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:16.461511 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:16.461543 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:16.461544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:16.461554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:16.461604 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:16.461906 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.462390 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.463023 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.463212 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.463261 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:16.471337 (+ 8076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:43645: BlockingRecv error: recv got EOF from 127.18.229.130:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 19:27:16 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:16 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:16.580354 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:16.575795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:37793)
0924 19:27:16.575934 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:16.575944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:16.576005 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:16.576175 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:16.576176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:16.576333 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:16.576381 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:16.576383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.577003 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.577003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:16.577511 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:16.577512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.577844 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.577844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:16.577899 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:16.578393 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:16.578398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:16.580258 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260924 19:27:16.580392 28524 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:16.575855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:37793 (local address 127.18.229.190:33763)
0924 19:27:16.575897 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:16.575903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:16.576005 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:27:16.576046 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:16.576047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:16.576059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:16.576165 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:16.576458 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.576938 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.577580 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:16.577761 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:16.577842 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:16.580332 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:37793: BlockingRecv error: recv got EOF from 127.18.229.131:37793 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 19:27:17 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:17 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:17.338330 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:17.333858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:46059)
0924 19:27:17.333972 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:17.333985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:17.334038 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:17.334172 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:17.334172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:17.334328 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:17.334376 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:17.334378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.334996 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.334997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:17.335512 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:17.335514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.335849 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.335850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:17.335914 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:17.336475 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:17.336480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:17.338233 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260924 19:27:17.338378 28528 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:17.333903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:46059 (local address 127.18.229.190:33763)
0924 19:27:17.334027 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:17.334028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:17.334036 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:27:17.334090 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:17.334090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:17.334102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:17.334157 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:17.334434 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.334908 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.335591 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.335787 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.335835 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:17.338309 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:46059: BlockingRecv error: recv got EOF from 127.18.229.129:46059 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
Sep 24 19:27:17 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:17 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:17.476521 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:17.471939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:57063)
0924 19:27:17.472132 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:17.472143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:17.472206 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:17.472336 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:17.472337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:17.472506 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:17.472557 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:17.472559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.473250 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.473251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:17.473810 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:17.473811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.474123 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.474124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:17.474162 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:17.474700 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:17.474704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:17.476463 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
W20260924 19:27:17.476649 28528 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:17.472075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57063 (local address 127.18.229.190:33763)
0924 19:27:17.472149 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:17.472152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:17.472190 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:27:17.472264 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:17.472265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:17.472278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:17.472318 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:17.472656 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.473159 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.473875 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.474060 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.474110 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:17.476570 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:57063: BlockingRecv error: recv got EOF from 127.18.229.130:57063 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 19:27:17 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:17 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:17.585422 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:17.580833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:35503)
0924 19:27:17.580962 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:17.580974 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:17.581042 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:17.581202 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:17.581203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:17.581371 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:17.581416 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:17.581417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.582074 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.582075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:17.582600 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:17.582602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.582969 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.582970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:17.583033 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:17.583518 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:17.583523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:17.585329 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260924 19:27:17.585469 28528 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:17.580926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:35503 (local address 127.18.229.190:33763)
0924 19:27:17.580996 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:17.580997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:17.581023 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:27:17.581128 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:17.581129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:17.581144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:17.581196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:17.581493 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.581989 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.582684 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:17.582885 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:17.582968 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:17.585405 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:35503: BlockingRecv error: recv got EOF from 127.18.229.131:35503 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 19:27:18 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:18 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:18.343356 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:18.338816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:49907)
0924 19:27:18.339043 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:18.339056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:18.339121 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:18.339284 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:18.339284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:18.339459 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:18.339504 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:18.339505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.340254 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.340255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:18.340786 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:18.340787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.341135 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.341136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:18.341174 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:18.341619 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:18.341624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:18.343288 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":61,"threads_started":1}
W20260924 19:27:18.343508 28532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:18.338859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:49907 (local address 127.18.229.190:33763)
0924 19:27:18.338990 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:18.338992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:18.339164 (+ 172us) server_negotiation.cc:407] Connection header received
0924 19:27:18.339190 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:18.339191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:18.339204 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:18.339273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:18.339595 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.340168 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.340860 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.341065 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.341123 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:18.343383 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:49907: BlockingRecv error: recv got EOF from 127.18.229.129:49907 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":44,"threads_started":1}
Sep 24 19:27:18 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:18 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:18.481643 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:18.476914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:55665)
0924 19:27:18.477080 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:18.477088 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:18.477139 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:18.477266 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:18.477266 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:18.477421 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:18.477466 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:18.477467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.478086 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.478086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:18.478638 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:18.478640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.479027 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.479028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:18.479090 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:18.479532 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:18.479537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:18.481576 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
W20260924 19:27:18.481756 28532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:18.476981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:55665 (local address 127.18.229.190:33763)
0924 19:27:18.477040 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:18.477042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:18.477139 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:27:18.477184 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:18.477185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:18.477195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:18.477247 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:18.477524 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.477997 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.478712 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.478901 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.478957 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:18.481673 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:55665: BlockingRecv error: recv got EOF from 127.18.229.130:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 19:27:18 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:18 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:18.590469 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:18.585877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:46053)
0924 19:27:18.586067 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:18.586077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:18.586142 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:18.586316 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:18.586317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:18.586525 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:18.586572 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:18.586574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.587197 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.587197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:18.587722 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:18.587724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.588193 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.588194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:18.588262 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:18.588729 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:18.588733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:18.590363 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
W20260924 19:27:18.590505 28532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:18.585966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:46053 (local address 127.18.229.190:33763)
0924 19:27:18.586035 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:18.586045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:18.586145 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:27:18.586184 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:18.586185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:18.586197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:18.586277 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:18.586643 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.587131 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.587794 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:18.588102 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:18.588185 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:18.590440 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:46053: BlockingRecv error: recv got EOF from 127.18.229.131:46053 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 19:27:19 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:19 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:19.348582 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:19.343844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:44263)
0924 19:27:19.343998 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:19.344008 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:19.344069 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:19.344300 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:19.344301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:19.344466 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:19.344512 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:19.344513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.345142 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.345143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:19.345658 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:19.345660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.345976 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.345977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:19.346015 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:19.346604 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:19.346610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:19.348519 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260924 19:27:19.348740 28536 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:19.343919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:44263 (local address 127.18.229.190:33763)
0924 19:27:19.344050 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:19.344052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:19.344067 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:27:19.344204 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:19.344205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:19.344218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:19.344281 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:19.344581 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.345074 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.345726 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.345914 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.345963 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:19.348639 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:44263: BlockingRecv error: recv got EOF from 127.18.229.129:44263 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
Sep 24 19:27:19 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:19 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:19.487013 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:19.482185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:55361)
0924 19:27:19.482370 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:19.482382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:19.482477 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:19.482623 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:19.482624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:19.482805 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:19.482851 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:19.482853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.483529 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.483531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:19.484062 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:19.484064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.484510 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.484511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:19.484579 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:19.485007 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:19.485011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:19.486920 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
W20260924 19:27:19.487058 28536 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:19.482279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:55361 (local address 127.18.229.190:33763)
0924 19:27:19.482355 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:19.482357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:19.482458 (+ 101us) server_negotiation.cc:407] Connection header received
0924 19:27:19.482536 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:19.482537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:19.482550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:19.482607 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:19.482934 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.483445 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.484220 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.484423 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.484509 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:19.487003 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:55361: BlockingRecv error: recv got EOF from 127.18.229.130:55361 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 19:27:19 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:19 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:19.595636 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:19.590905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:44957)
0924 19:27:19.591078 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:19.591088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:19.591172 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:19.591322 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:19.591323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:19.591500 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:19.591544 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:19.591546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.592256 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.592257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:19.593070 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:19.593073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.593437 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.593437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:19.593494 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:19.593920 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:19.593924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:19.595547 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
W20260924 19:27:19.595701 28536 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:19.590990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:44957 (local address 127.18.229.190:33763)
0924 19:27:19.591057 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:19.591060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:19.591155 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:27:19.591227 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:19.591228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:19.591243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:19.591316 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:19.591623 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.592175 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.593162 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:19.593354 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:19.593436 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:19.595638 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:44957: BlockingRecv error: recv got EOF from 127.18.229.131:44957 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 19:27:20 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:20 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:20.353595 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:20.349117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:41759)
0924 19:27:20.349338 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:20.349350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:20.349416 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:20.349589 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:20.349590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:20.349753 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:20.349797 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:20.349799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.350410 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.350411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:20.350929 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:20.350930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.351241 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.351242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:20.351281 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:20.351715 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:20.351719 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:20.353484 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
W20260924 19:27:20.353641 28540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:20.349152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:41759 (local address 127.18.229.190:33763)
0924 19:27:20.349290 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:20.349292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:20.349442 (+ 150us) server_negotiation.cc:407] Connection header received
0924 19:27:20.349473 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:20.349474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:20.349488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:20.349566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:20.349866 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.350348 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.350998 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.351182 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.351229 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:20.353570 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:41759: BlockingRecv error: recv got EOF from 127.18.229.129:41759 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
Sep 24 19:27:20 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:20 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:20.492338 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:20.487617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:37395)
0924 19:27:20.487763 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:20.487776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:20.487850 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:20.487988 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:20.487989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:20.488254 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:20.488299 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:20.488301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.489121 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.489122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:20.489655 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:20.489657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.489971 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.489972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:20.490010 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:20.490431 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:20.490435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:20.492273 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20260924 19:27:20.492455 28540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:20.487744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:37395 (local address 127.18.229.190:33763)
0924 19:27:20.487809 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:20.487811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:20.487833 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:27:20.487909 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:20.487910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:20.487920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:20.487973 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:20.488409 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.489049 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.489723 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.489908 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.489961 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:20.492349 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:37395: BlockingRecv error: recv got EOF from 127.18.229.130:37395 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 19:27:20 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:20 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:20.600709 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:20.596102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:39307)
0924 19:27:20.596300 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:20.596309 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:20.596401 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:20.596611 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:20.596613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:20.596788 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:20.596830 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:20.596832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.597496 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.597497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:20.598048 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:20.598050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.598411 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.598412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:20.598474 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:20.598886 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:20.598890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:20.600604 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
W20260924 19:27:20.600746 28540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:20.596160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:39307 (local address 127.18.229.190:33763)
0924 19:27:20.596207 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:20.596209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:20.596367 (+ 158us) server_negotiation.cc:407] Connection header received
0924 19:27:20.596507 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:20.596507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:20.596524 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:20.596604 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:20.596916 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.597416 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.598143 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:20.598332 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:20.598410 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:20.600682 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:39307: BlockingRecv error: recv got EOF from 127.18.229.131:39307 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 19:27:21 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:21 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:21.359037 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:21.354251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:41489)
0924 19:27:21.354409 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:21.354494 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:21.354571 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:21.354739 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:21.354739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:21.354920 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:21.354966 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:21.354968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.355684 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.355686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:21.356294 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:21.356296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.356681 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.356682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:21.356742 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:21.357203 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:21.357208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:21.358927 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
W20260924 19:27:21.359076 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:21.354286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:41489 (local address 127.18.229.190:33763)
0924 19:27:21.354426 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:21.354428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:21.354561 (+ 133us) server_negotiation.cc:407] Connection header received
0924 19:27:21.354627 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:21.354628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:21.354643 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:21.354721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:21.355070 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.355589 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.356377 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.356593 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.356679 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:21.359007 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:41489: BlockingRecv error: recv got EOF from 127.18.229.129:41489 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
Sep 24 19:27:21 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:21 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:21.501417 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:21.492811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:36871)
0924 19:27:21.492956 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:21.492966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:21.493026 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:21.493203 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:21.493204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:21.493377 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:21.493422 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:21.493424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.494082 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.494083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:21.494609 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:21.494611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.495001 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.495002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:21.495068 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:21.499534 (+ 4466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:21.499539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:21.501359 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
W20260924 19:27:21.501538 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:21.492867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:36871 (local address 127.18.229.190:33763)
0924 19:27:21.492936 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:21.492938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:21.493031 (+ 93us) server_negotiation.cc:407] Connection header received
0924 19:27:21.493105 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:21.493106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:21.493121 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:21.493185 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:21.493503 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.493997 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.494694 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.494897 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.494987 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:21.501460 (+ 6473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:36871: BlockingRecv error: recv got EOF from 127.18.229.130:36871 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 19:27:21 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:21 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.131@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:21.605794 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:21.601120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.131:39905)
0924 19:27:21.601293 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:21.601310 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:21.601373 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:21.601485 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:21.601486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:21.601650 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:21.601695 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:21.601696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.602321 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.602322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:21.602970 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:21.602973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.603439 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.603440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:21.603482 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:21.604022 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:21.604027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:21.605736 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
W20260924 19:27:21.605890 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:21.601211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:39905 (local address 127.18.229.190:33763)
0924 19:27:21.601286 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:21.601288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:21.601377 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:27:21.601418 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:21.601419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:21.601430 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:21.601480 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:21.601761 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.602252 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.603040 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:21.603365 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:21.603428 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:21.605824 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.131:39905: BlockingRecv error: recv got EOF from 127.18.229.131:39905 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 19:27:22 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:22 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.129@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:22.364050 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.359726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.129:37911)
0924 19:27:22.359867 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:22.359877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:22.359934 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:22.360089 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:22.360090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:22.360278 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:22.360327 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.360328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.360949 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.360950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.361484 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.361486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.361805 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.361805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.361844 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.362279 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:22.362283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:22.363973 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260924 19:27:22.364202 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:22.359795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:37911 (local address 127.18.229.190:33763)
0924 19:27:22.359924 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:22.359926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:22.359933 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:27:22.359991 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:22.359991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:22.360002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:22.360067 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:22.360394 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.360883 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.361554 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.361741 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.361790 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.364074 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.129:37911: BlockingRecv error: recv got EOF from 127.18.229.129:37911 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Sep 24 19:27:22 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:27:22 dist-test-slave-r8cc krb5kdc[27490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.229.130@KRBTEST.COM for oryx/127.18.229.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:27:22.506441 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.501955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:33763 (local address 127.18.229.130:45713)
0924 19:27:22.502097 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:22.502107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:22.502170 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:22.502295 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:22.502295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:22.502447 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:22.502497 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.502498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.503120 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.503120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.503645 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.503646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.503962 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.503963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.504000 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.504497 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:22.504502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:22.506332 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.229.190:33763: Server oryx/127.18.229.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260924 19:27:22.506489 28548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:22.502010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:45713 (local address 127.18.229.190:33763)
0924 19:27:22.502086 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:22.502088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:22.502173 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:27:22.502216 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:22.502217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:22.502228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:22.502278 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:22.502562 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.503055 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.503716 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.503900 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.503954 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.506420 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.229.130:45713: BlockingRecv error: recv got EOF from 127.18.229.130:45713 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:27:22.530017 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28074
I20260924 19:27:22.534735 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28225
I20260924 19:27:22.538730 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28358
I20260924 19:27:22.542788 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28003
2026-09-24T19:27:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17420 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T19:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:22Z Disabled control of system clock
I20260924 19:27:22.570264 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44963
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:27:22.649439 28553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:22.649633 28553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:22.649650 28553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:22.650774 28553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:22.650815 28553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:22.650827 28553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:22.650840 28553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:22.650888 28553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:22.652211 28553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:44963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:22.652459 28553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:22.652745 28553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:22.657550 28561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:22.657564 28553 server_base.cc:1034] running on GCE node
W20260924 19:27:22.657527 28558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:22.657533 28559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:22.657985 28553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:22.658217 28553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:22.659361 28553 hybrid_clock.cc:648] HybridClock initialized: now 1790278042659331 us; error 37 us; skew 500 ppm
I20260924 19:27:22.660506 28553 webserver.cc:460] Webserver started at http://127.18.229.190:39353/ using document root <none> and password file <none>
I20260924 19:27:22.660701 28553 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:22.660740 28553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:22.660825 28553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:22.661752 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "d7ea73f1e2774434818ea38b83657855"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "fc278247f28f03044735901acac5394f"
server_key_iv: "31aa8cc29fc8a4cf1e63ca5c11fde5ed"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.662027 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "d7ea73f1e2774434818ea38b83657855"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "fc278247f28f03044735901acac5394f"
server_key_iv: "31aa8cc29fc8a4cf1e63ca5c11fde5ed"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.663233 28553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:27:22.663913 28567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.664068 28553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:22.664156 28553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "d7ea73f1e2774434818ea38b83657855"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "fc278247f28f03044735901acac5394f"
server_key_iv: "31aa8cc29fc8a4cf1e63ca5c11fde5ed"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.664225 28553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:22.678583 28553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:22.678838 28553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:22.678979 28553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:22.682637 28553 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:44963
I20260924 19:27:22.682684 28619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:44963 every 8 connection(s)
I20260924 19:27:22.682981 28553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:27:22.683619 28620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:22.684542 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28553
I20260924 19:27:22.684633 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:27:22.684824 19350 external_mini_cluster.cc:1428] Setting key d60da86dd8a5292e6d1fba30e0ef1365
I20260924 19:27:22.686676 28620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855: Bootstrap starting.
I20260924 19:27:22.687256 28620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:22.687569 28620 log.cc:824] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:22.688280 28620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855: No bootstrap required, opened a new log
I20260924 19:27:22.689972 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.685314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.18.229.190:44963)
0924 19:27:22.685505 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:22.685511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:22.685536 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:27:22.685923 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:22.685928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:22.685983 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:22.686097 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:22.686320 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.687197 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.688262 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.688682 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.688774 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.689370 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:22.689373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:22.689373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:22.689521 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:22.689554 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:22.689564 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:22.689613 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:22.689704 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:27:22.689779 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260924 19:27:22.690569 28620 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ea73f1e2774434818ea38b83657855" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44963 } }
I20260924 19:27:22.690712 28620 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:22.690754 28620 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ea73f1e2774434818ea38b83657855, State: Initialized, Role: FOLLOWER
I20260924 19:27:22.690881 28620 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ea73f1e2774434818ea38b83657855" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44963 } }
I20260924 19:27:22.690955 28620 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:27:22.690997 28620 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:27:22.691044 28620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:22.691632 28620 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ea73f1e2774434818ea38b83657855" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44963 } }
I20260924 19:27:22.691741 28620 leader_election.cc:302] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ea73f1e2774434818ea38b83657855; no voters:
I20260924 19:27:22.691898 28620 leader_election.cc:288] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:27:22.691926 28625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:22.692046 28625 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [term 1 LEADER]: Becoming Leader. State: Replica: d7ea73f1e2774434818ea38b83657855, State: Running, Role: LEADER
I20260924 19:27:22.692174 28620 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:27:22.692206 28625 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ea73f1e2774434818ea38b83657855" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44963 } }
I20260924 19:27:22.692642 28626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7ea73f1e2774434818ea38b83657855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ea73f1e2774434818ea38b83657855" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44963 } } }
I20260924 19:27:22.692727 28626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:22.692900 28627 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ea73f1e2774434818ea38b83657855. Latest consensus state: current_term: 1 leader_uuid: "d7ea73f1e2774434818ea38b83657855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ea73f1e2774434818ea38b83657855" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 44963 } } }
I20260924 19:27:22.692941 28634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:22.693010 28627 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:22.693506 28634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:22.694921 28634 catalog_manager.cc:1347] Generated new cluster ID: c78f42b865d94757a1bfbe4d6b179ef7
I20260924 19:27:22.694971 28634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:27:22.704334 28634 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:27:22.704756 28634 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:22.707249 28634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7ea73f1e2774434818ea38b83657855: Generated new TSK 0
I20260924 19:27:22.707378 28634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:27:22.710880 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44963
--builtin_ntp_servers=127.18.229.148:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:27:22.792723 28644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:22.792999 28644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:22.793035 28644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:22.794296 28644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:22.794334 28644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:22.794360 28644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:27:22.795667 28644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:22.795890 28644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:22.796241 28644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:22.801083 28649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:22.801046 28650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:22.801038 28652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:22.801420 28644 server_base.cc:1034] running on GCE node
I20260924 19:27:22.801625 28644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:22.801874 28644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:22.803012 28644 hybrid_clock.cc:648] HybridClock initialized: now 1790278042802997 us; error 32 us; skew 500 ppm
I20260924 19:27:22.804280 28644 webserver.cc:460] Webserver started at http://127.18.229.129:42225/ using document root <none> and password file <none>
I20260924 19:27:22.804526 28644 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:22.804570 28644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:22.804710 28644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:22.805737 28644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "ce80c95cd5e94cf5a93f81553778697f"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "809b665e3639fbe99e2243626a87be88"
server_key_iv: "b32f9470300a0387fe83133db685e5f4"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.806082 28644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "ce80c95cd5e94cf5a93f81553778697f"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "809b665e3639fbe99e2243626a87be88"
server_key_iv: "b32f9470300a0387fe83133db685e5f4"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.807371 28644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:22.808091 28658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.808290 28644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:22.808387 28644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "ce80c95cd5e94cf5a93f81553778697f"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "809b665e3639fbe99e2243626a87be88"
server_key_iv: "b32f9470300a0387fe83133db685e5f4"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.808462 28644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:22.820483 28644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:22.820711 28644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:22.820806 28644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:22.820966 28644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:22.821206 28644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:22.821236 28644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.821256 28644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:22.821276 28644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.826706 28644 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:36991
I20260924 19:27:22.826743 28771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:36991 every 8 connection(s)
I20260924 19:27:22.827057 28644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:27:22.828786 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28644
I20260924 19:27:22.828881 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:27:22.829101 19350 external_mini_cluster.cc:1428] Setting key aab14c741c13d1c3b408694840ad94a2
I20260924 19:27:22.830035 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44963
--builtin_ntp_servers=127.18.229.148:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:27:22.833051 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.827763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:40989 (local address 127.18.229.190:44963)
0924 19:27:22.827812 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:22.827815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:22.828050 (+ 235us) server_negotiation.cc:407] Connection header received
0924 19:27:22.828573 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:22.828574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:22.828584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:22.828625 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:22.829469 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.830008 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.831498 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.831814 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.831892 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.832743 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:22.832744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:22.832744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:22.832749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:22.832766 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:22.832771 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:22.832823 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:22.832898 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:27:22.832926 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:27:22.833199 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.827765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44963 (local address 127.18.229.129:40989)
0924 19:27:22.827939 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:22.827969 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:22.828482 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:22.828715 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:22.828720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:22.828731 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:22.829373 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.829378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.830116 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.830117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.831408 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.831411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.831900 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.831901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.832001 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.832612 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:27:22.832612 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:22.832639 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:27:22.832660 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:22.832837 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:22.832841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:22.832843 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:27:22.832886 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:27:22.832960 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 19:27:22.833711 28772 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44963
I20260924 19:27:22.833814 28772 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:22.833976 28772 heartbeater.cc:511] Master 127.18.229.190:44963 requested a full tablet report, sending...
I20260924 19:27:22.834467 28584 ts_manager.cc:194] Registered new tserver with Master: ce80c95cd5e94cf5a93f81553778697f (127.18.229.129:36991)
I20260924 19:27:22.835073 28584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.129:40989
W20260924 19:27:22.910208 28775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:22.910406 28775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:22.910426 28775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:22.911649 28775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:22.911687 28775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:22.911712 28775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:27:22.913183 28775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:22.913430 28775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:22.913738 28775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:22.918540 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:22.918547 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:22.918593 28775 server_base.cc:1034] running on GCE node
W20260924 19:27:22.918556 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:22.918957 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:22.919216 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:22.920367 28775 hybrid_clock.cc:648] HybridClock initialized: now 1790278042920354 us; error 28 us; skew 500 ppm
I20260924 19:27:22.921486 28775 webserver.cc:460] Webserver started at http://127.18.229.130:40225/ using document root <none> and password file <none>
I20260924 19:27:22.921730 28775 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:22.921795 28775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:22.921926 28775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:22.922936 28775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "46e976530e92499a874cee5f5ef827b9"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "b68e31fcdcf385e458496b31b089782c"
server_key_iv: "1f7cac86a50fe0dfd4929260ce5907da"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.923310 28775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "46e976530e92499a874cee5f5ef827b9"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "b68e31fcdcf385e458496b31b089782c"
server_key_iv: "1f7cac86a50fe0dfd4929260ce5907da"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.924600 28775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:22.925261 28789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.925414 28775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:22.925504 28775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "46e976530e92499a874cee5f5ef827b9"
format_stamp: "Formatted at 2026-09-24 19:27:22 on dist-test-slave-r8cc"
server_key: "b68e31fcdcf385e458496b31b089782c"
server_key_iv: "1f7cac86a50fe0dfd4929260ce5907da"
server_key_version: "encryptionkey@0"
I20260924 19:27:22.925581 28775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:22.936962 28775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:22.937234 28775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:22.937376 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:22.937597 28775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:22.937888 28775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:22.937937 28775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.937968 28775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:22.937995 28775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:22.943405 28775 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:37081
I20260924 19:27:22.943465 28902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:37081 every 8 connection(s)
I20260924 19:27:22.943816 28775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:27:22.945439 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28775
I20260924 19:27:22.945535 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:27:22.945760 19350 external_mini_cluster.cc:1428] Setting key 9ca41bd6f6d9afce7263411b9aa35206
I20260924 19:27:22.946738 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44963
--builtin_ntp_servers=127.18.229.148:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:27:22.949733 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.944515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:41383 (local address 127.18.229.190:44963)
0924 19:27:22.944555 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:22.944557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:22.944785 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:27:22.945263 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:22.945265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:22.945274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:22.945323 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:22.946176 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.946718 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.948274 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.948593 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.948668 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.949441 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:22.949442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:22.949443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:22.949447 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:22.949464 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:22.949468 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:22.949536 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:22.949586 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:27:22.949616 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:27:22.949831 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:22.944530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44963 (local address 127.18.229.130:41383)
0924 19:27:22.944691 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:22.944721 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:22.945170 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:22.945402 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:22.945407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:22.945418 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:22.946055 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.946059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.946801 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.946802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.948188 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:22.948191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:22.948671 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:22.948672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:22.948776 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:22.949311 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:27:22.949311 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:22.949337 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:27:22.949357 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:22.949530 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:22.949533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:22.949536 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:22.949586 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:27:22.949659 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260924 19:27:22.950260 28903 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44963
I20260924 19:27:22.950356 28903 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:22.950536 28903 heartbeater.cc:511] Master 127.18.229.190:44963 requested a full tablet report, sending...
I20260924 19:27:22.950876 28584 ts_manager.cc:194] Registered new tserver with Master: 46e976530e92499a874cee5f5ef827b9 (127.18.229.130:37081)
I20260924 19:27:22.951412 28584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.130:41383
W20260924 19:27:23.026460 28906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:23.026651 28906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:23.026671 28906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:23.027755 28906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:23.027786 28906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:23.027842 28906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:27:23.029253 28906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:23.029444 28906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:23.029685 28906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:23.034381 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.034405 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.034407 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:23.034572 28906 server_base.cc:1034] running on GCE node
I20260924 19:27:23.034834 28906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:23.035069 28906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:23.036176 28906 hybrid_clock.cc:648] HybridClock initialized: now 1790278043036169 us; error 35 us; skew 500 ppm
I20260924 19:27:23.037369 28906 webserver.cc:460] Webserver started at http://127.18.229.131:43173/ using document root <none> and password file <none>
I20260924 19:27:23.037611 28906 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:23.037676 28906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:23.037806 28906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:23.038823 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "840c40c7bc5f4a149454c1e5f5374ff1"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "9d4a008e7c728b0c735ac305d9dda763"
server_key_iv: "8919ccd46808ef2336076b148338146a"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.039175 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "840c40c7bc5f4a149454c1e5f5374ff1"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "9d4a008e7c728b0c735ac305d9dda763"
server_key_iv: "8919ccd46808ef2336076b148338146a"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.040488 28906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:23.041160 28920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.041317 28906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:23.041405 28906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "840c40c7bc5f4a149454c1e5f5374ff1"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "9d4a008e7c728b0c735ac305d9dda763"
server_key_iv: "8919ccd46808ef2336076b148338146a"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.041482 28906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:23.056298 28906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:23.056557 28906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:23.056689 28906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:23.056886 28906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:23.057232 28906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:23.057276 28906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.057304 28906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:23.057333 28906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.062678 28906 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:39763
I20260924 19:27:23.062737 29033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:39763 every 8 connection(s)
I20260924 19:27:23.063038 28906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:27:23.068214 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.063680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:51469 (local address 127.18.229.190:44963)
0924 19:27:23.063760 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.063763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.063967 (+ 204us) server_negotiation.cc:407] Connection header received
0924 19:27:23.064502 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.064503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.064513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.064558 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.065330 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:23.066076 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:23.067042 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:23.067227 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:23.067278 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:23.067963 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:23.067964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:23.067964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:23.067968 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:23.067982 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:23.067986 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:23.068040 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:23.068092 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 19:27:23.068153 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:27:23.068401 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.063655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:44963 (local address 127.18.229.131:51469)
0924 19:27:23.063843 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:23.063889 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:23.064407 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:23.064606 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:23.064610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:23.064618 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:23.065222 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:23.065228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:23.066151 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:23.066151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:23.066960 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:23.066962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:23.067290 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:23.067291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:23.067391 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:23.067812 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:27:23.067813 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:23.067838 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:27:23.067859 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:23.068043 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:23.068046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:23.068048 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:27:23.068094 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:27:23.068209 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260924 19:27:23.068821 29034 heartbeater.cc:347] Connected to a master server at 127.18.229.190:44963
I20260924 19:27:23.068917 29034 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:23.069085 29034 heartbeater.cc:511] Master 127.18.229.190:44963 requested a full tablet report, sending...
I20260924 19:27:23.069415 28584 ts_manager.cc:194] Registered new tserver with Master: 840c40c7bc5f4a149454c1e5f5374ff1 (127.18.229.131:39763)
I20260924 19:27:23.069774 28584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.131:51469
I20260924 19:27:23.072235 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 28906
I20260924 19:27:23.072332 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:27:23.072525 19350 external_mini_cluster.cc:1428] Setting key b7602aa45658a1265970e92ff3f78d49
I20260924 19:27:23.073246 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:27:23.074970 28623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:23.074472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.18.229.190:44963)
0924 19:27:23.074539 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.074541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.074650 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:27:23.074724 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.074725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.074735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.074789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.074909 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42098: BlockingRecv error: recv got EOF from 127.0.0.1:42098 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:27:23.075045 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28644
I20260924 19:27:23.079886 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28775
I20260924 19:27:23.084359 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28906
I20260924 19:27:23.088370 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 28553
2026-09-24T19:27:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (544 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T19:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:23Z Disabled control of system clock
I20260924 19:27:23.114295 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34819
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:27:23.193017 29045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:23.193253 29045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:23.193290 29045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:23.194398 29045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:23.194468 29045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:23.194489 29045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:23.194509 29045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:23.194538 29045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:23.195876 29045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:23.196153 29045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:23.196424 29045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:27:23.201187 29045 server_base.cc:1034] running on GCE node
W20260924 19:27:23.201134 29053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.201119 29050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.201159 29051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:23.201598 29045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:23.201884 29045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:23.203042 29045 hybrid_clock.cc:648] HybridClock initialized: now 1790278043203029 us; error 39 us; skew 500 ppm
I20260924 19:27:23.204227 29045 webserver.cc:460] Webserver started at http://127.18.229.190:41635/ using document root <none> and password file <none>
I20260924 19:27:23.204442 29045 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:23.204501 29045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:23.204627 29045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:23.205631 29045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "eb92d6b9e5d90827e539402f11438fb5"
server_key_iv: "8377d36126954993d24fe15cf96d2c2c"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.205976 29045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "eb92d6b9e5d90827e539402f11438fb5"
server_key_iv: "8377d36126954993d24fe15cf96d2c2c"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.207178 29045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:23.207829 29059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.207986 29045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:23.208081 29045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "eb92d6b9e5d90827e539402f11438fb5"
server_key_iv: "8377d36126954993d24fe15cf96d2c2c"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.208168 29045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:23.211254 29045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:23.211486 29045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:23.211642 29045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:23.215306 29045 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:34819
I20260924 19:27:23.215334 29111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:34819 every 8 connection(s)
I20260924 19:27:23.215652 29045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:27:23.216310 29112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:23.218217 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29045
I20260924 19:27:23.218324 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:27:23.218536 19350 external_mini_cluster.cc:1428] Setting key c1b8fc93cff3220dcf136a053b69a59f
I20260924 19:27:23.219298 29112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155: Bootstrap starting.
I20260924 19:27:23.219915 29112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:23.220311 29112 log.cc:824] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:23.221149 29112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155: No bootstrap required, opened a new log
I20260924 19:27:23.221364 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.219075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.18.229.190:34819)
0924 19:27:23.219247 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.219254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.219276 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:27:23.219817 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.219827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.219879 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.220045 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.220840 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:23.220844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:23.220844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:23.220849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:23.220894 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:23.220908 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:23.220995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:23.221127 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 19:27:23.221186 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 19:27:23.223425 29112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34819 } }
I20260924 19:27:23.223573 29112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:23.223618 29112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd8cb15228f4bf48cc6c3a0b85cd155, State: Initialized, Role: FOLLOWER
I20260924 19:27:23.223731 29112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34819 } }
I20260924 19:27:23.223819 29112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:27:23.223860 29112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:27:23.223907 29112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:23.224472 29112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34819 } }
I20260924 19:27:23.224581 29112 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd8cb15228f4bf48cc6c3a0b85cd155; no voters:
I20260924 19:27:23.224725 29112 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:27:23.224752 29117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:23.224903 29117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd8cb15228f4bf48cc6c3a0b85cd155, State: Running, Role: LEADER
I20260924 19:27:23.224990 29117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34819 } }
I20260924 19:27:23.225016 29112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:27:23.225309 29119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd8cb15228f4bf48cc6c3a0b85cd155. Latest consensus state: current_term: 1 leader_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34819 } } }
I20260924 19:27:23.225309 29118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd8cb15228f4bf48cc6c3a0b85cd155" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34819 } } }
I20260924 19:27:23.225525 29119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:23.225551 29118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155 [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:23.225906 29126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:23.226419 29126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:23.227855 29126 catalog_manager.cc:1347] Generated new cluster ID: e98659b68c504244b84bc024dd31cd9f
I20260924 19:27:23.227895 29126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:27:23.239090 29126 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:27:23.239248 29126 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:23.245529 29126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd8cb15228f4bf48cc6c3a0b85cd155: Generated new TSK 0
I20260924 19:27:23.245685 29126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:27:23.254580 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34819
--builtin_ntp_servers=127.18.229.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:27:23.337502 29136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:23.337721 29136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:23.337759 29136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:23.338919 29136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:23.338980 29136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:23.339025 29136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:27:23.340443 29136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:23.340687 29136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:23.340962 29136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:27:23.347007 29136 server_base.cc:1034] running on GCE node
W20260924 19:27:23.346971 29142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.347075 29144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.346964 29141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:23.347349 29136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:23.347605 29136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:23.348759 29136 hybrid_clock.cc:648] HybridClock initialized: now 1790278043348747 us; error 29 us; skew 500 ppm
I20260924 19:27:23.350024 29136 webserver.cc:460] Webserver started at http://127.18.229.129:45055/ using document root <none> and password file <none>
I20260924 19:27:23.350250 29136 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:23.350312 29136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:23.350461 29136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:23.351531 29136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "8cd60a63d1964b7eb748f5f34b5e7b67"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "6ebd603b753f1c2a7051d000afdad2cf"
server_key_iv: "82ff9cab5a52ff91aaa188c1fc16fb08"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.351888 29136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "8cd60a63d1964b7eb748f5f34b5e7b67"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "6ebd603b753f1c2a7051d000afdad2cf"
server_key_iv: "82ff9cab5a52ff91aaa188c1fc16fb08"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.353300 29136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:23.354084 29150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.354260 29136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:23.354355 29136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "8cd60a63d1964b7eb748f5f34b5e7b67"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "6ebd603b753f1c2a7051d000afdad2cf"
server_key_iv: "82ff9cab5a52ff91aaa188c1fc16fb08"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.354452 29136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:23.357823 29136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:23.358079 29136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:23.358215 29136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:23.358448 29136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:23.358737 29136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:23.358785 29136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.358824 29136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:23.358855 29136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.364741 29136 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:40001
I20260924 19:27:23.364789 29263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:40001 every 8 connection(s)
I20260924 19:27:23.365116 29136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:27:23.367798 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.365781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:49467 (local address 127.18.229.190:34819)
0924 19:27:23.365858 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.365861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.366083 (+ 222us) server_negotiation.cc:407] Connection header received
0924 19:27:23.366542 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.366543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.366548 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.366600 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.367611 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:23.367612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:23.367613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:23.367616 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:23.367631 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:23.367634 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:23.367678 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:23.367735 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:27:23.367746 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:27:23.367988 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.365768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34819 (local address 127.18.229.129:49467)
0924 19:27:23.365966 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:23.366001 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:23.366457 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:23.366648 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:23.366651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:23.366659 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:23.367486 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:27:23.367486 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:23.367513 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:27:23.367539 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:23.367676 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:23.367679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:23.367681 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:27:23.367732 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:27:23.367775 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260924 19:27:23.368474 29264 heartbeater.cc:347] Connected to a master server at 127.18.229.190:34819
I20260924 19:27:23.368551 29264 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:23.368708 29264 heartbeater.cc:511] Master 127.18.229.190:34819 requested a full tablet report, sending...
I20260924 19:27:23.369177 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29136
I20260924 19:27:23.369208 29076 ts_manager.cc:194] Registered new tserver with Master: 8cd60a63d1964b7eb748f5f34b5e7b67 (127.18.229.129:40001)
I20260924 19:27:23.369272 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:27:23.369441 19350 external_mini_cluster.cc:1428] Setting key 44974a115f1536005a7bfa2a85f0f8e5
I20260924 19:27:23.370296 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34819
--builtin_ntp_servers=127.18.229.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:27:23.450078 29267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:23.450265 29267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:23.450287 29267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:23.451411 29267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:23.451448 29267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:23.451494 29267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:27:23.452893 29267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:23.453128 29267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:23.453436 29267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:23.458447 29275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.458456 29273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.458467 29272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:23.458674 29267 server_base.cc:1034] running on GCE node
I20260924 19:27:23.458891 29267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:23.459090 29267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:23.460179 29267 hybrid_clock.cc:648] HybridClock initialized: now 1790278043460167 us; error 25 us; skew 500 ppm
I20260924 19:27:23.461308 29267 webserver.cc:460] Webserver started at http://127.18.229.130:43287/ using document root <none> and password file <none>
I20260924 19:27:23.461505 29267 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:23.461545 29267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:23.461629 29267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:23.462625 29267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "b4a3252dd3b34c11999381d1fb776af0"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "f5d735641033bcd341f0555759d9fac7"
server_key_iv: "cbe7de7156bc606d47b0d6fb62cd0ed5"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.462913 29267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "b4a3252dd3b34c11999381d1fb776af0"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "f5d735641033bcd341f0555759d9fac7"
server_key_iv: "cbe7de7156bc606d47b0d6fb62cd0ed5"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.464079 29267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:27:23.464846 29281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.465008 29267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:23.465103 29267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "b4a3252dd3b34c11999381d1fb776af0"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "f5d735641033bcd341f0555759d9fac7"
server_key_iv: "cbe7de7156bc606d47b0d6fb62cd0ed5"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.465173 29267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:23.468398 29267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:23.468648 29267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:23.468768 29267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:23.468966 29267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:23.469257 29267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:23.469322 29267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.469358 29267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:23.469384 29267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.474674 29267 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41277
I20260924 19:27:23.474706 29394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41277 every 8 connection(s)
I20260924 19:27:23.474996 29267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:27:23.477427 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.475600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:55079 (local address 127.18.229.190:34819)
0924 19:27:23.475673 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.475676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.475844 (+ 168us) server_negotiation.cc:407] Connection header received
0924 19:27:23.476300 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.476301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.476306 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.476357 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.477253 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:23.477254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:23.477255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:23.477258 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:23.477270 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:23.477274 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:23.477314 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:23.477368 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:27:23.477379 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:27:23.477592 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.475577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34819 (local address 127.18.229.130:55079)
0924 19:27:23.475743 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:23.475772 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:23.476225 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:23.476403 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:23.476405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:23.476413 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:23.477141 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:27:23.477142 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:23.477164 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:27:23.477186 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:23.477317 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:23.477319 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:23.477321 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:27:23.477366 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:27:23.477405 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260924 19:27:23.478022 29395 heartbeater.cc:347] Connected to a master server at 127.18.229.190:34819
I20260924 19:27:23.478091 29395 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:23.478235 29395 heartbeater.cc:511] Master 127.18.229.190:34819 requested a full tablet report, sending...
I20260924 19:27:23.478546 29076 ts_manager.cc:194] Registered new tserver with Master: b4a3252dd3b34c11999381d1fb776af0 (127.18.229.130:41277)
I20260924 19:27:23.484758 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29267
I20260924 19:27:23.484860 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:27:23.485075 19350 external_mini_cluster.cc:1428] Setting key dffd1f4e3a1996f96bda7f7d73f3d0ed
I20260924 19:27:23.485921 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34819
--builtin_ntp_servers=127.18.229.148:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:27:23.565572 29398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:23.565766 29398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:23.565788 29398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:23.566920 29398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:23.566957 29398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:23.566999 29398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:27:23.568384 29398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:23.568607 29398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:23.568879 29398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:23.573544 29406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:23.573540 29403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:23.573642 29398 server_base.cc:1034] running on GCE node
W20260924 19:27:23.573563 29404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:23.573951 29398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:23.574199 29398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:23.575348 29398 hybrid_clock.cc:648] HybridClock initialized: now 1790278043575339 us; error 35 us; skew 500 ppm
I20260924 19:27:23.576598 29398 webserver.cc:460] Webserver started at http://127.18.229.131:43105/ using document root <none> and password file <none>
I20260924 19:27:23.576826 29398 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:23.576887 29398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:23.577013 29398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:23.577982 29398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "60debe000b0e47bfbe10171da19b700f"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "7bfabec42b87207bf22890073d01dc94"
server_key_iv: "9ace545035115deb61189f15bbebd076"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.578328 29398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "60debe000b0e47bfbe10171da19b700f"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "7bfabec42b87207bf22890073d01dc94"
server_key_iv: "9ace545035115deb61189f15bbebd076"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.579608 29398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:27:23.580402 29412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.580582 29398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:27:23.580674 29398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "60debe000b0e47bfbe10171da19b700f"
format_stamp: "Formatted at 2026-09-24 19:27:23 on dist-test-slave-r8cc"
server_key: "7bfabec42b87207bf22890073d01dc94"
server_key_iv: "9ace545035115deb61189f15bbebd076"
server_key_version: "encryptionkey@0"
I20260924 19:27:23.580751 29398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:23.583921 29398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:23.584214 29398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:23.584343 29398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:23.584544 29398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:23.584834 29398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:23.584882 29398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.584920 29398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:23.584947 29398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:23.590765 29398 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:37249
I20260924 19:27:23.590827 29525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:37249 every 8 connection(s)
I20260924 19:27:23.591131 29398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:27:23.593917 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.591802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:58751 (local address 127.18.229.190:34819)
0924 19:27:23.591865 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.591867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.592082 (+ 215us) server_negotiation.cc:407] Connection header received
0924 19:27:23.592584 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.592585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.592590 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.592650 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.593759 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:23.593760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:23.593760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:27:23.593763 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:23.593771 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:27:23.593774 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:23.593814 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:23.593857 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:27:23.593867 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:27:23.594094 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:23.591785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34819 (local address 127.18.229.131:58751)
0924 19:27:23.591963 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:23.592000 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:23.592490 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:23.592705 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:23.592708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:23.592715 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:23.593636 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:27:23.593636 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:23.593662 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:27:23.593694 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:23.593812 (+ 118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:23.593815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:23.593817 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:27:23.593864 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:27:23.593907 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260924 19:27:23.594532 29526 heartbeater.cc:347] Connected to a master server at 127.18.229.190:34819
I20260924 19:27:23.594627 29526 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:23.594728 29526 heartbeater.cc:511] Master 127.18.229.190:34819 requested a full tablet report, sending...
I20260924 19:27:23.595032 29076 ts_manager.cc:194] Registered new tserver with Master: 60debe000b0e47bfbe10171da19b700f (127.18.229.131:37249)
I20260924 19:27:23.600175 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29398
I20260924 19:27:23.600255 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:27:23.600410 19350 external_mini_cluster.cc:1428] Setting key 51d094ee01ad0a51d802ba2d172bf6be
I20260924 19:27:23.601235 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:27:23.602969 29115 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:27:23.602446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.18.229.190:34819)
0924 19:27:23.602501 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:23.602524 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:23.602632 (+ 108us) server_negotiation.cc:407] Connection header received
0924 19:27:23.602706 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:23.602707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:23.602712 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:23.602777 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:23.602900 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38784: BlockingRecv error: recv got EOF from 127.0.0.1:38784 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:27:23.603003 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29136
I20260924 19:27:23.607836 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29267
I20260924 19:27:23.611863 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29398
I20260924 19:27:23.615760 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29045
2026-09-24T19:27:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (527 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:27:23 dist-test-slave-r8cc krb5kdc[29536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:27:23 dist-test-slave-r8cc krb5kdc[29536](info): set up 2 sockets
Sep 24 19:27:23 dist-test-slave-r8cc krb5kdc[29536](info): commencing operation
krb5kdc: starting...
W20260924 19:27:24.628854 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:27:24 dist-test-slave-r8cc krb5kdc[29536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:27:24.738540 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38849
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:38849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:24.816655 29549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:24.816862 29549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:24.816891 29549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:24.817960 29549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:27:24.818056 29549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:24.818079 29549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:24.818100 29549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:27:24.818125 29549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:27:24.819525 29549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:38849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:24.819790 29549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:24.819993 29549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:24.824451 29555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:24.824452 29557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:24.824549 29549 server_base.cc:1034] running on GCE node
W20260924 19:27:24.824476 29554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:24.824883 29549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:24.825150 29549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:24.826288 29549 hybrid_clock.cc:648] HybridClock initialized: now 1790278044826279 us; error 28 us; skew 500 ppm
Sep 24 19:27:24 dist-test-slave-r8cc krb5kdc[29536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278044, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:24.828716 29549 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:27:24.829107 29549 webserver.cc:460] Webserver started at http://127.18.229.190:34091/ using document root <none> and password file <none>
I20260924 19:27:24.829289 29549 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:24.829326 29549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:24.829456 29549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:24.830448 29549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "956600d58b9b40b0a7e2fde0d7ff094a"
format_stamp: "Formatted at 2026-09-24 19:27:24 on dist-test-slave-r8cc"
server_key: "51d6e18668f663684ec19ef1b612116d"
server_key_iv: "f4b2f8da66fccbfd8a6aece340ba710f"
server_key_version: "encryptionkey@0"
I20260924 19:27:24.830739 29549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "956600d58b9b40b0a7e2fde0d7ff094a"
format_stamp: "Formatted at 2026-09-24 19:27:24 on dist-test-slave-r8cc"
server_key: "51d6e18668f663684ec19ef1b612116d"
server_key_iv: "f4b2f8da66fccbfd8a6aece340ba710f"
server_key_version: "encryptionkey@0"
I20260924 19:27:24.832005 29549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:27:24.832839 29564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:24.833000 29549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:24.833098 29549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "956600d58b9b40b0a7e2fde0d7ff094a"
format_stamp: "Formatted at 2026-09-24 19:27:24 on dist-test-slave-r8cc"
server_key: "51d6e18668f663684ec19ef1b612116d"
server_key_iv: "f4b2f8da66fccbfd8a6aece340ba710f"
server_key_version: "encryptionkey@0"
I20260924 19:27:24.833171 29549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:24.880041 29549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:24.882522 29549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:24.882683 29549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:24.886605 29549 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:38849
I20260924 19:27:24.886643 29616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:38849 every 8 connection(s)
I20260924 19:27:24.887038 29549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:27:24.887982 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:24.890265 29617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a: Bootstrap starting.
I20260924 19:27:24.890785 29617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:24.891021 29617 log.cc:824] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:24.891675 29617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a: No bootstrap required, opened a new log
I20260924 19:27:24.893036 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29549
I20260924 19:27:24.893141 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:27:24.893321 19350 external_mini_cluster.cc:1428] Setting key 7bfccbac42dc494264ebb4db9c383b47
I20260924 19:27:24.893354 29617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38849 } }
I20260924 19:27:24.893561 29617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:24.893610 29617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956600d58b9b40b0a7e2fde0d7ff094a, State: Initialized, Role: FOLLOWER
I20260924 19:27:24.893767 29617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38849 } }
I20260924 19:27:24.893983 29617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:27:24.894048 29617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:27:24.894107 29617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:24.894930 29617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38849 } }
I20260924 19:27:24.895051 29617 leader_election.cc:302] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 956600d58b9b40b0a7e2fde0d7ff094a; no voters:
I20260924 19:27:24.895236 29617 leader_election.cc:288] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:27:24.895263 29622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:24.895361 29622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [term 1 LEADER]: Becoming Leader. State: Replica: 956600d58b9b40b0a7e2fde0d7ff094a, State: Running, Role: LEADER
I20260924 19:27:24.895439 29622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38849 } }
I20260924 19:27:24.895486 29617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:27:24.895736 29622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38849 } } }
I20260924 19:27:24.895886 29622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:24.896066 29622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 956600d58b9b40b0a7e2fde0d7ff094a. Latest consensus state: current_term: 1 leader_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "956600d58b9b40b0a7e2fde0d7ff094a" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 38849 } } }
I20260924 19:27:24.896214 29622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a [sys.catalog]: This master's current role is: LEADER
I20260924 19:27:24.896436 29629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:27:24.897037 29629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:27:24.898691 29629 catalog_manager.cc:1347] Generated new cluster ID: a6b91efee5a044b183102a6ad2937c45
I20260924 19:27:24.898746 29629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:27:24 dist-test-slave-r8cc krb5kdc[29536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:24.903759 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:24.893845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.18.229.190:38849)
0924 19:27:24.894068 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:24.894077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:24.894110 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:27:24.894236 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:24.894242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:24.894284 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:24.894414 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:24.894993 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:24.896391 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:24.897465 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:24.898039 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:24.898136 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:24.900654 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:24.900658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:24.900659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:24.900672 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:24.902330 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:24.902703 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:24.902706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:24.902706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:24.902800 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:24.903006 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:24.903007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:24.903007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:24.903230 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:24.903400 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:24.903468 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:27:24.903612 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260924 19:27:24.910574 29629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:27:24.911223 29629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:27:24.917235 29629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a: Generated new TSK 0
I20260924 19:27:24.917387 29629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:27:24.944947 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38849
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:25.036444 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:25.036631 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:25.036654 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:25.037832 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:25.037879 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:25.037910 29643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:27:25.039335 29643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:25.039557 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:25.039798 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:25.045104 29651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:25.045122 29648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:25.045104 29649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:25.045468 29643 server_base.cc:1034] running on GCE node
I20260924 19:27:25.045655 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:25.045924 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:25.047070 29643 hybrid_clock.cc:648] HybridClock initialized: now 1790278045047059 us; error 26 us; skew 500 ppm
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[29536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278045, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:25.049715 29643 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:27:25.050200 29643 webserver.cc:460] Webserver started at http://127.18.229.129:41115/ using document root <none> and password file <none>
I20260924 19:27:25.050447 29643 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:25.050495 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:25.050588 29643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:25.051523 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "55f27bfce2fe4c61950651b57fd452f6"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "666de27556d8df6e8e94680232ec97e7"
server_key_iv: "8e78ebe8b482a81f8d15ec1aa68929d6"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.051807 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "55f27bfce2fe4c61950651b57fd452f6"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "666de27556d8df6e8e94680232ec97e7"
server_key_iv: "8e78ebe8b482a81f8d15ec1aa68929d6"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.053102 29643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:27:25.053740 29658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.053912 29643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:27:25.053980 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "55f27bfce2fe4c61950651b57fd452f6"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "666de27556d8df6e8e94680232ec97e7"
server_key_iv: "8e78ebe8b482a81f8d15ec1aa68929d6"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.054033 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:25.072784 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:25.074479 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:25.074587 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:25.074749 29643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:25.075028 29643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:25.075059 29643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.075117 29643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:25.075142 29643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.080863 29643 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:44189
I20260924 19:27:25.080932 29771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:44189 every 8 connection(s)
I20260924 19:27:25.081240 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[29536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278045, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:25.089676 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.081962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:41205 (local address 127.18.229.190:38849)
0924 19:27:25.082020 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.082022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.082227 (+ 205us) server_negotiation.cc:407] Connection header received
0924 19:27:25.082708 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.082709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.082720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.082772 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:25.083666 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.084167 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.085075 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.085256 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.085305 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.087219 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:25.087220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:25.087221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:25.087228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:25.088621 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.088948 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.088949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.088949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.088994 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.089181 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.089182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.089182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.089275 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:25.089324 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.089540 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.089609 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:27:25.089833 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.081948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38849 (local address 127.18.229.129:41205)
0924 19:27:25.082104 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.082144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.082625 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.082833 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.082836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.083397 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:25.083592 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.083596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.084230 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.084231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.085005 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.085007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.085321 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.085322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.085418 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.085852 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:25.085860 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:25.087136 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:25.088692 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:25.088697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:25.088698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:25.088893 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:25.089044 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:25.089045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:25.089045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:25.089133 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:25.089323 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:25.089327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:25.089431 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.089516 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.089624 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260924 19:27:25.090466 29772 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38849
I20260924 19:27:25.090467 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29643
I20260924 19:27:25.090575 29772 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:25.090579 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:27:25.090771 29772 heartbeater.cc:511] Master 127.18.229.190:38849 requested a full tablet report, sending...
I20260924 19:27:25.090803 19350 external_mini_cluster.cc:1428] Setting key 4c47c85f7cf2f544a4be422818c6bdcd
I20260924 19:27:25.091356 29580 ts_manager.cc:194] Registered new tserver with Master: 55f27bfce2fe4c61950651b57fd452f6 (127.18.229.129:44189)
I20260924 19:27:25.091969 29580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:41205
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:27:25.112787 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38849
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:25.192607 29777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:25.192806 29777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:25.192830 29777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:25.193948 29777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:25.193989 29777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:25.194015 29777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:27:25.195472 29777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:25.195700 29777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:25.195911 29777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:25.200928 29785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:25.200954 29783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:25.200917 29782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:25.201038 29777 server_base.cc:1034] running on GCE node
I20260924 19:27:25.201550 29777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:25.201757 29777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:25.202934 29777 hybrid_clock.cc:648] HybridClock initialized: now 1790278045202910 us; error 34 us; skew 500 ppm
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[29536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278045, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:25.205572 29777 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:27:25.206002 29777 webserver.cc:460] Webserver started at http://127.18.229.130:37249/ using document root <none> and password file <none>
I20260924 19:27:25.206248 29777 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:25.206311 29777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:25.206467 29777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:25.207490 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "b93ac3dc9734468a94c0d9e4664d8185"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "cc33f82798fabf2e9a3e6d195ff72b59"
server_key_iv: "d53550bba9c7862a78b105262ae1467c"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.207839 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "b93ac3dc9734468a94c0d9e4664d8185"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "cc33f82798fabf2e9a3e6d195ff72b59"
server_key_iv: "d53550bba9c7862a78b105262ae1467c"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.209259 29777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:27:25.209996 29792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.210156 29777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:25.210248 29777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "b93ac3dc9734468a94c0d9e4664d8185"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "cc33f82798fabf2e9a3e6d195ff72b59"
server_key_iv: "d53550bba9c7862a78b105262ae1467c"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.210327 29777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:25.222644 29777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:25.224388 29777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:25.224529 29777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:25.224726 29777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:25.225011 29777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:25.225060 29777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.225101 29777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:25.225129 29777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.231639 29777 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:34801
I20260924 19:27:25.231699 29905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:34801 every 8 connection(s)
I20260924 19:27:25.231989 29777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:27:25.237179 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29777
I20260924 19:27:25.237264 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:27:25.237512 19350 external_mini_cluster.cc:1428] Setting key e619d20db2d09504b014473375dd0173
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[29536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278045, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:25.241308 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.232775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:55213 (local address 127.18.229.190:38849)
0924 19:27:25.232845 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.232847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.232999 (+ 152us) server_negotiation.cc:407] Connection header received
0924 19:27:25.233466 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.233467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.233477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.233525 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:25.234446 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.234966 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.235970 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.236276 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.236332 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.238530 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:25.238531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:25.238532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:25.238539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:25.239919 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.240399 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.240400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.240400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.240446 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.240710 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.240711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.240711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.240801 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:25.240846 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.241171 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.241247 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:27:25.241557 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.232739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38849 (local address 127.18.229.130:55213)
0924 19:27:25.232889 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.232922 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.233382 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.233583 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.233586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.234186 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:25.234367 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.234372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.235059 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.235060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.235884 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.235886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.236342 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.236343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.236447 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.236863 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:25.236870 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:25.238434 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:25.239994 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:25.239999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:25.240002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:25.240331 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:25.240504 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:25.240505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:25.240506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:25.240647 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:25.240855 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:25.240860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:25.241026 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.241136 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.241298 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260924 19:27:25.242007 29906 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38849
I20260924 19:27:25.242111 29906 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:25.242274 29906 heartbeater.cc:511] Master 127.18.229.190:38849 requested a full tablet report, sending...
I20260924 19:27:25.242672 29580 ts_manager.cc:194] Registered new tserver with Master: b93ac3dc9734468a94c0d9e4664d8185 (127.18.229.130:34801)
I20260924 19:27:25.243026 29580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:55213
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:27:25.262209 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38849
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:27:25.342285 29911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:27:25.342475 29911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:27:25.342502 29911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:27:25.343626 29911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:27:25.343667 29911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:27:25.343715 29911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:27:25.345170 29911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:27:25.345389 29911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:27:25.345598 29911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:27:25.350406 29916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:27:25.350425 29917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:25.350548 29911 server_base.cc:1034] running on GCE node
W20260924 19:27:25.350435 29919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:27:25.350797 29911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:27:25.351040 29911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:27:25.352169 29911 hybrid_clock.cc:648] HybridClock initialized: now 1790278045352156 us; error 28 us; skew 500 ppm
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[29536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278045, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:27:25.354415 29911 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:27:25.354844 29911 webserver.cc:460] Webserver started at http://127.18.229.131:38563/ using document root <none> and password file <none>
I20260924 19:27:25.355038 29911 fs_manager.cc:359] Metadata directory not provided
I20260924 19:27:25.355078 29911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:27:25.355161 29911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:27:25.356093 29911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "b2475d97d0114741810c8daefc7ddc75"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "d86c14d8c998e62a1416c3fb03982bce"
server_key_iv: "036961e9c0f9062cad247cb3967ae4b2"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.356462 29911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "b2475d97d0114741810c8daefc7ddc75"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "d86c14d8c998e62a1416c3fb03982bce"
server_key_iv: "036961e9c0f9062cad247cb3967ae4b2"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.357651 29911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:27:25.358361 29926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.358507 29911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:27:25.358582 29911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "b2475d97d0114741810c8daefc7ddc75"
format_stamp: "Formatted at 2026-09-24 19:27:25 on dist-test-slave-r8cc"
server_key: "d86c14d8c998e62a1416c3fb03982bce"
server_key_iv: "036961e9c0f9062cad247cb3967ae4b2"
server_key_version: "encryptionkey@0"
I20260924 19:27:25.358630 29911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:27:25.366673 29911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:27:25.368363 29911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:27:25.368468 29911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:27:25.368618 29911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:27:25.368896 29911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:27:25.368927 29911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.368966 29911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:27:25.368991 29911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:27:25.374445 29911 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:45965
I20260924 19:27:25.374518 30039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:45965 every 8 connection(s)
I20260924 19:27:25.374825 29911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:27:25.376219 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 29911
I20260924 19:27:25.376302 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:27:25.376533 19350 external_mini_cluster.cc:1428] Setting key f2463ef2e3b2cc003e3ce9d129b201e4
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[29536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278045, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:27:25.383826 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.375578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:34147 (local address 127.18.229.190:38849)
0924 19:27:25.375632 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.375634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.375834 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:27:25.376483 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.376485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.376508 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.376590 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:25.377749 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.378233 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.379189 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.379374 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.379429 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.381530 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:25.381531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:25.381531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:25.381538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:25.382694 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.383033 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.383034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.383034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.383078 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.383267 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.383267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.383267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.383359 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:25.383407 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.383658 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.383738 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:27:25.384003 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.375545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:38849 (local address 127.18.229.131:34147)
0924 19:27:25.375715 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.375756 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.376369 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.376686 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.376690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.377466 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:27:25.377674 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.377680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.378306 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.378307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.379108 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.379110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.379454 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.379455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.379557 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.380017 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:27:25.380025 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:27:25.381450 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:27:25.382760 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:25.382765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:25.382767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:25.382973 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:25.383130 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:27:25.383131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:27:25.383131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:27:25.383219 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:27:25.383407 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:27:25.383410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:27:25.383522 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.383628 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.383776 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260924 19:27:25.384486 30040 heartbeater.cc:347] Connected to a master server at 127.18.229.190:38849
I20260924 19:27:25.384595 30040 heartbeater.cc:464] Registering TS with master...
I20260924 19:27:25.384774 30040 heartbeater.cc:511] Master 127.18.229.190:38849 requested a full tablet report, sending...
I20260924 19:27:25.385110 29580 ts_manager.cc:194] Registered new tserver with Master: b2475d97d0114741810c8daefc7ddc75 (127.18.229.131:45965)
I20260924 19:27:25.385430 29580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:34147
I20260924 19:27:25.388190 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:27:25.395008 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.389377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.18.229.190:38849)
0924 19:27:25.389430 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.389431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.389561 (+ 130us) server_negotiation.cc:407] Connection header received
0924 19:27:25.389635 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.389636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.389644 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.389705 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:27:25.390249 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.390721 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.391407 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.391613 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.391692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.392813 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:27:25.392814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:27:25.392815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:27:25.392822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:27:25.393881 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.394218 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.394219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.394219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.394272 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:27:25.394459 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:27:25.394460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:27:25.394460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:27:25.394597 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:27:25.394702 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.394826 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.394903 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:27:25.396464 29580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:27:25.398239 29580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:27:25.404367 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.400204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:44189 (local address 127.0.0.1:37546)
0924 19:27:25.400494 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.400502 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.400540 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.400750 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.400753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.400756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:25.400817 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.400819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.402253 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.402254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.403193 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.403195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.404044 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.404045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.404141 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.404157 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.404231 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.404290 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":46,"thread_start_us":55,"threads_started":1}
I20260924 19:27:25.404377 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.400376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.18.229.129:44189)
0924 19:27:25.400561 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.400565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.400575 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:27:25.400616 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.400618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.400635 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.400720 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:25.400954 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.402126 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.403273 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.403937 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.404012 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.404066 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.404250 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.404299 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260924 19:27:25.404664 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.400399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:34801 (local address 127.0.0.1:33390)
0924 19:27:25.400658 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.400669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.400722 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.400957 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.400958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.400960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:25.401000 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.401001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.402240 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.402241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.403419 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.403422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.404417 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.404419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.404490 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.404495 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.404550 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.404595 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260924 19:27:25.404886 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.400203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45965 (local address 127.0.0.1:59436)
0924 19:27:25.400300 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.400338 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.400395 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.401028 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.401028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.401031 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:25.401087 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.401089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.402081 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.402082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.403420 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.403422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.404380 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.404381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.404642 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.404647 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.404708 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.404824 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":43,"thread_start_us":50,"threads_started":1}
I20260924 19:27:25.404942 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.400500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.18.229.130:34801)
0924 19:27:25.400664 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.400668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.400711 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:27:25.400786 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.400788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.400804 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.400878 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:25.401076 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.402142 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.403504 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.404266 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.404332 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.404383 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.404828 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.404868 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260924 19:27:25.405171 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.400262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.18.229.131:45965)
0924 19:27:25.400858 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.400863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.400872 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:25.400905 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.400906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.400920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.401160 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:25.401219 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.401979 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.403528 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.404240 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.404297 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.404347 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.404823 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.404865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":71,"threads_started":1}
I20260924 19:27:25.405200 29706 tablet_service.cc:1467] Processing CreateTablet for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 (DEFAULT_TABLE table=test-table [id=d433808585be41a09984a7071789b640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:25.405580 29706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:25.405591 29840 tablet_service.cc:1467] Processing CreateTablet for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 (DEFAULT_TABLE table=test-table [id=d433808585be41a09984a7071789b640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:25.405887 29840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:25.405838 29974 tablet_service.cc:1467] Processing CreateTablet for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 (DEFAULT_TABLE table=test-table [id=d433808585be41a09984a7071789b640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:27:25.406109 29974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:27:25.408025 30054 tablet_bootstrap.cc:492] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6: Bootstrap starting.
I20260924 19:27:25.408810 30054 tablet_bootstrap.cc:654] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:25.409099 30056 tablet_bootstrap.cc:492] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Bootstrap starting.
I20260924 19:27:25.409413 30054 log.cc:824] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:25.409739 30056 tablet_bootstrap.cc:654] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:25.410228 30054 tablet_bootstrap.cc:492] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6: No bootstrap required, opened a new log
I20260924 19:27:25.410285 30056 log.cc:824] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:25.410303 30054 ts_tablet_manager.cc:1397] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:25.410615 30055 tablet_bootstrap.cc:492] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Bootstrap starting.
I20260924 19:27:25.410929 30056 tablet_bootstrap.cc:492] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: No bootstrap required, opened a new log
I20260924 19:27:25.411008 30056 ts_tablet_manager.cc:1397] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:25.411223 30055 tablet_bootstrap.cc:654] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Neither blocks nor log segments found. Creating new log.
I20260924 19:27:25.411859 30055 log.cc:824] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Log is configured to *not* fsync() on all Append() calls
I20260924 19:27:25.412639 30055 tablet_bootstrap.cc:492] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: No bootstrap required, opened a new log
I20260924 19:27:25.412703 30055 ts_tablet_manager.cc:1397] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:27:25.412596 30054 raft_consensus.cc:348] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.412755 30054 raft_consensus.cc:374] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:25.412849 30054 raft_consensus.cc:729] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f27bfce2fe4c61950651b57fd452f6, State: Initialized, Role: FOLLOWER
I20260924 19:27:25.412873 30056 raft_consensus.cc:348] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.412971 30054 consensus_queue.cc:260] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.413043 30056 raft_consensus.cc:374] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:25.413087 30056 raft_consensus.cc:729] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93ac3dc9734468a94c0d9e4664d8185, State: Initialized, Role: FOLLOWER
I20260924 19:27:25.413228 30054 ts_tablet_manager.cc:1428] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:27:25.413237 30056 consensus_queue.cc:260] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.413352 29772 heartbeater.cc:503] Master 127.18.229.190:38849 was elected leader, sending a full tablet report...
I20260924 19:27:25.413503 30056 ts_tablet_manager.cc:1428] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:27:25.413517 29906 heartbeater.cc:503] Master 127.18.229.190:38849 was elected leader, sending a full tablet report...
I20260924 19:27:25.414678 30055 raft_consensus.cc:348] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.414788 30055 raft_consensus.cc:374] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:27:25.414806 30055 raft_consensus.cc:729] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2475d97d0114741810c8daefc7ddc75, State: Initialized, Role: FOLLOWER
I20260924 19:27:25.414871 30055 consensus_queue.cc:260] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.415066 30055 ts_tablet_manager.cc:1428] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:27:25.415128 30040 heartbeater.cc:503] Master 127.18.229.190:38849 was elected leader, sending a full tablet report...
I20260924 19:27:25.447597 30061 raft_consensus.cc:482] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:27:25.447710 30061 raft_consensus.cc:504] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.448019 30061 leader_election.cc:288] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f27bfce2fe4c61950651b57fd452f6 (127.18.229.129:44189), b2475d97d0114741810c8daefc7ddc75 (127.18.229.131:45965)
I20260924 19:27:25.450791 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.448273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57893 (local address 127.18.229.131:45965)
0924 19:27:25.448318 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.448320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.448332 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:27:25.448414 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.448415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.448425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.448495 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:25.448705 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.449192 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.450103 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.450449 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.450499 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.450528 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.450678 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.450723 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:27:25.450793 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.448228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45965 (local address 127.18.229.130:57893)
0924 19:27:25.448290 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.448302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.448343 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.448557 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.448558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.448560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:25.448634 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.448636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.449268 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.449269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.450007 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.450009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.450527 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.450528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.450603 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.450606 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.450675 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.450714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 19:27:25.450827 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.448282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:47401 (local address 127.18.229.129:44189)
0924 19:27:25.448335 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.448336 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.448492 (+ 156us) server_negotiation.cc:407] Connection header received
0924 19:27:25.448571 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.448572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.448583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.448642 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:27:25.448789 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.449212 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.450093 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.450456 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.450544 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.450551 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.450723 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.450761 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:27:25.450889 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.448247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:44189 (local address 127.18.229.130:47401)
0924 19:27:25.448424 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.448435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:27:25.448490 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:27:25.448649 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:27:25.448650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:27:25.448651 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:27:25.448728 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.448729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.449303 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.449304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.450005 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:27:25.450006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.450556 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.450556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:27:25.450611 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.450614 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:27:25.450715 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:27:25.450748 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260924 19:27:25.451128 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b4903dc6d74a2ca6d9d0514ab8fc77" candidate_uuid: "b93ac3dc9734468a94c0d9e4664d8185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2475d97d0114741810c8daefc7ddc75" is_pre_election: true
I20260924 19:27:25.451314 29994 raft_consensus.cc:2454] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b93ac3dc9734468a94c0d9e4664d8185 in term 0.
I20260924 19:27:25.451320 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b4903dc6d74a2ca6d9d0514ab8fc77" candidate_uuid: "b93ac3dc9734468a94c0d9e4664d8185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f27bfce2fe4c61950651b57fd452f6" is_pre_election: true
I20260924 19:27:25.451478 29726 raft_consensus.cc:2454] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b93ac3dc9734468a94c0d9e4664d8185 in term 0.
I20260924 19:27:25.451570 29796 leader_election.cc:302] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2475d97d0114741810c8daefc7ddc75, b93ac3dc9734468a94c0d9e4664d8185; no voters:
I20260924 19:27:25.451689 30061 raft_consensus.cc:2781] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:27:25.451742 30061 raft_consensus.cc:482] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:27:25.451774 30061 raft_consensus.cc:3037] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:25.452487 30061 raft_consensus.cc:504] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.452625 30061 leader_election.cc:288] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [CANDIDATE]: Term 1 election: Requested vote from peers 55f27bfce2fe4c61950651b57fd452f6 (127.18.229.129:44189), b2475d97d0114741810c8daefc7ddc75 (127.18.229.131:45965)
I20260924 19:27:25.452785 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b4903dc6d74a2ca6d9d0514ab8fc77" candidate_uuid: "b93ac3dc9734468a94c0d9e4664d8185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2475d97d0114741810c8daefc7ddc75"
I20260924 19:27:25.452839 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b4903dc6d74a2ca6d9d0514ab8fc77" candidate_uuid: "b93ac3dc9734468a94c0d9e4664d8185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f27bfce2fe4c61950651b57fd452f6"
I20260924 19:27:25.452863 29994 raft_consensus.cc:3037] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:25.452927 29726 raft_consensus.cc:3037] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:27:25.453548 29994 raft_consensus.cc:2454] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b93ac3dc9734468a94c0d9e4664d8185 in term 1.
I20260924 19:27:25.453661 29726 raft_consensus.cc:2454] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b93ac3dc9734468a94c0d9e4664d8185 in term 1.
I20260924 19:27:25.453737 29796 leader_election.cc:302] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2475d97d0114741810c8daefc7ddc75, b93ac3dc9734468a94c0d9e4664d8185; no voters:
I20260924 19:27:25.453817 30061 raft_consensus.cc:2781] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:27:25.453950 30061 raft_consensus.cc:686] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 1 LEADER]: Becoming Leader. State: Replica: b93ac3dc9734468a94c0d9e4664d8185, State: Running, Role: LEADER
I20260924 19:27:25.454056 30061 consensus_queue.cc:237] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } }
I20260924 19:27:25.454900 29581 catalog_manager.cc:5495] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 reported cstate change: term changed from 0 to 1, leader changed from <none> to b93ac3dc9734468a94c0d9e4664d8185 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "b93ac3dc9734468a94c0d9e4664d8185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b93ac3dc9734468a94c0d9e4664d8185" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 } health_report { overall_health: UNKNOWN } } }
I20260924 19:27:25.461293 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.459150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.18.229.130:34801)
0924 19:27:25.459202 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:27:25.459204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:27:25.459213 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:27:25.459253 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:27:25.459254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:27:25.459267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:27:25.459326 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:27:25.459428 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.459855 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.460592 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:27:25.460764 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:27:25.460827 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:27:25.460974 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:27:25.461076 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:27:25.461135 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:27:25.461184 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 19:27:25.461199 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:27:25.462878 29994 raft_consensus.cc:1261] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 1 FOLLOWER]: Refusing update from remote peer b93ac3dc9734468a94c0d9e4664d8185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:25.462878 29726 raft_consensus.cc:1261] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P 55f27bfce2fe4c61950651b57fd452f6 [term 1 FOLLOWER]: Refusing update from remote peer b93ac3dc9734468a94c0d9e4664d8185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:27:25.463192 30064 consensus_queue.cc:1035] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f27bfce2fe4c61950651b57fd452f6" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:25.463290 30061 consensus_queue.cc:1035] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2475d97d0114741810c8daefc7ddc75" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:27:25.465283 30067 mvcc.cc:205] Tried to move back new op lower bound from 7332978874213691392 to 7332978874180616192. Current Snapshot: MvccSnapshot[applied={T|T < 7332978874213691392}]
I20260924 19:27:25.465616 30069 mvcc.cc:205] Tried to move back new op lower bound from 7332978874213691392 to 7332978874180616192. Current Snapshot: MvccSnapshot[applied={T|T < 7332978874213691392}]
I20260924 19:27:25.468014 29580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:25.468089 29580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:27:25.468792 29580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 956600d58b9b40b0a7e2fde0d7ff094a: Sending DeleteTablet for 3 replicas of tablet 47b4903dc6d74a2ca6d9d0514ab8fc77
I20260924 19:27:25.469094 29706 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:25 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:37546
I20260924 19:27:25.469148 29974 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:25 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:59436
I20260924 19:27:25.469275 29840 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:27:25 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:33390
I20260924 19:27:25.469390 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29643
I20260924 19:27:25.470597 30079 tablet_replica.cc:330] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: stopping tablet replica
I20260924 19:27:25.470698 30079 raft_consensus.cc:2229] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:27:25.470808 30079 raft_consensus.cc:2258] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:25.471221 30079 ts_tablet_manager.cc:1905] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:25.472906 30079 ts_tablet_manager.cc:1918] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:25.472978 30079 log.cc:1196] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/47b4903dc6d74a2ca6d9d0514ab8fc77
I20260924 19:27:25.473261 30079 ts_tablet_manager.cc:1939] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b93ac3dc9734468a94c0d9e4664d8185: Deleting consensus metadata
I20260924 19:27:25.473903 29565 catalog_manager.cc:4841] TS b93ac3dc9734468a94c0d9e4664d8185 (127.18.229.130:34801): tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 (table test-table [id=d433808585be41a09984a7071789b640]) successfully deleted
I20260924 19:27:25.476204 30078 tablet_replica.cc:330] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: stopping tablet replica
I20260924 19:27:25.476315 30078 raft_consensus.cc:2229] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:27:25.476416 30078 raft_consensus.cc:2258] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:27:25.476794 30078 ts_tablet_manager.cc:1905] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:27:25.478669 30078 ts_tablet_manager.cc:1918] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:27:25.478740 30078 log.cc:1196] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/47b4903dc6d74a2ca6d9d0514ab8fc77
I20260924 19:27:25.478991 30078 ts_tablet_manager.cc:1939] T 47b4903dc6d74a2ca6d9d0514ab8fc77 P b2475d97d0114741810c8daefc7ddc75: Deleting consensus metadata
I20260924 19:27:25.479583 29568 catalog_manager.cc:4841] TS b2475d97d0114741810c8daefc7ddc75 (127.18.229.131:45965): tablet 47b4903dc6d74a2ca6d9d0514ab8fc77 (table test-table [id=d433808585be41a09984a7071789b640]) successfully deleted
W20260924 19:27:25.480212 29565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:44189 (error 108)
I20260924 19:27:25.480688 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:27:25.480561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:44189 (local address 127.0.0.1:37562)
0924 19:27:25.480614 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:27:25.480655 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:27:25.480772 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29777
W20260924 19:27:25.484208 29565 catalog_manager.cc:4584] TS 55f27bfce2fe4c61950651b57fd452f6 (127.18.229.129:44189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47b4903dc6d74a2ca6d9d0514ab8fc77: Network error: Client connection negotiation failed: client connection to 127.18.229.129:44189: connect: Connection refused (error 111)
I20260924 19:27:25.485773 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29911
I20260924 19:27:25.490128 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 29549
2026-09-24T19:27:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1874 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[30083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[30083](info): set up 2 sockets
Sep 24 19:27:25 dist-test-slave-r8cc krb5kdc[30083](info): commencing operation
krb5kdc: starting...
W20260924 19:27:26.525065 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:27:26 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:27:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/rangeradmin_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/rangeradmin_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/rangerlookup_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/rangerlookup_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
I20260924 19:27:26.678262 19350 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2KIbLR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/postgres -l logfile start
2026-09-24 19:27:28.957 UTC [30113] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:27:28.957 UTC [30113] LOG: listening on IPv4 address "127.18.229.148", port 36027
2026-09-24 19:27:28.961 UTC [30113] LOG: listening on Unix socket "/tmp/.s.PGSQL.36027"
2026-09-24 19:27:28.995 UTC [30116] LOG: database system was shut down at 2026-09-24 19:27:27 UTC
2026-09-24 19:27:28.999 UTC [30113] LOG: database system is ready to accept connections
I20260924 19:27:29.957441 19350 mini_postgres.cc:96] Postgres bound to 36027
2026-09-24 19:27:29.963 UTC [30124] FATAL: database "slave" does not exist
127.18.229.148:36027 - accepting connections
I20260924 19:27:29.963912 19350 mini_ranger.cc:161] Starting Ranger...
I20260924 19:27:29.978060 19350 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:27:30.398800 19350 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:27:30.398882 19350 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-admin
2026-09-24 19:27:30,667 [I] DB FLAVOR :POSTGRES
2026-09-24 19:27:30,667 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:27:30,667 [I] Checking connection..
2026-09-24 19:27:30,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:27:30,932 [I] Checking connection passed.
2026-09-24 19:27:30,933 [I] --------- Verifying version history table ---------
2026-09-24 19:27:30,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:27:31,191 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:27:31,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:27:31,448 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:27:31,448 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:27:31,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:27:31.704 UTC [30188] WARNING: there is no transaction in progress
2026-09-24 19:27:31,711 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:27:31,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:27:31,944 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:27:31,945 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:27:31,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:27:32,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:27:32,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r8cc.c.gcp-upstream.internal','N') ;"
2026-09-24 19:27:32,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:27:32,855 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:27:32,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:27:33,094 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:27:33,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:27:33,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:27:33,572 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:27:33,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:27:34.480 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.496 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.512 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.527 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.542 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.561 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.614 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.620 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.634 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.644 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.654 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.667 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.675 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.681 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.687 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:34.693 UTC [30296] WARNING: there is no transaction in progress
2026-09-24 19:27:35,276 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:27:35,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:27:35,494 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:27:35,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:27:35,723 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:27:35,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:27:35,967 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:27:35,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:27:36,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:27:36,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r8cc.c.gcp-upstream.internal';"
2026-09-24 19:27:36,662 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:27:36,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:27:36,893 [I] DB_PATCHES have already been applied
I20260924 19:27:36.899436 19350 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-admin:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2KIbLR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:27:36.899507 19350 mini_ranger.cc:193] Using host: 127.18.229.148
I20260924 19:27:36.902490 19350 mini_ranger.cc:227] Ranger admin URL: http://127.18.229.148:52833
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:27:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:27:38.012122 19350 mini_ranger.cc:160] Time spent starting Ranger: real 8.048s user 0.000s sys 0.009s
Sep 24, 2026 7:27:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:27:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:27:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:27:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:27:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:27:59.619843 19350 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/rangerkms_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/rangerkms_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.229.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.229.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:27:59 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278079, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:27:59.697870 19350 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:27:59.708463 19350 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:28:00.502847 19350 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:28:00.502912 19350 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms
I20260924 19:28:00.502925 19350 mini_ranger_kms.cc:227] Using postgres at 127.18.229.148:36027
2026-09-24 19:28:00,740 [I] DB FLAVOR :POSTGRES
2026-09-24 19:28:00,740 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:28:00,740 [I] Checking connection
2026-09-24 19:28:00,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:28:00,967 [I] connection success
2026-09-24 19:28:00,967 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:28:00,967 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:28:00,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:28:01,223 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:28:01,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:28:01,224 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:28:01,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:36027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:28:01,481 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:28:01.943809 19350 mini_ranger_kms.cc:310] Created kms service
I20260924 19:28:02.107295 19350 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:28:02.148741 19350 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:28:02 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278079, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:02.520632 19350 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:28:02.520742 19350 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms:/tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:28:02.520766 19350 mini_ranger_kms.cc:240] Using host: 127.18.229.148
I20260924 19:28:02.523972 19350 mini_ranger_kms.cc:276] Ranger KMS PID: 30476
I20260924 19:28:02.524123 19350 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.229.148:45735
Sep 24, 2026 7:28:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:28:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:28:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:28:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:28:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:28:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:28:04.746145 19350 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.048s user 0.000s sys 0.221s
I20260924 19:28:04.746255 19350 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:28:04.746340 19350 mini_ranger_kms.cc:401] 127.18.229.148:45735/kms/v1/keys
Sep 24 19:28:04 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278084, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.229.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:28:05 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278084, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.229.148@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
Sep 24 19:28:05 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278084, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.229.148@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
Sep 24 19:28:07 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:28:07.888818 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:40817
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:40817
--ranger_config_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:45735/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:07.968981 30527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:07.969172 30527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:07.969192 30527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:07.970326 30527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:28:07.970366 30527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:07.970381 30527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:07.970392 30527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:28:07.970404 30527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:28:07.971952 30527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.229.148:45735/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:40817
--ranger_config_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:07.972208 30527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:07.972448 30527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:07.978421 30533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:07.978502 30527 server_base.cc:1034] running on GCE node
W20260924 19:28:07.978446 30535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:07.978628 30532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:07.978988 30527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:07.979305 30527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:07.980468 30527 hybrid_clock.cc:648] HybridClock initialized: now 1790278087980456 us; error 31 us; skew 500 ppm
Sep 24 19:28:07 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278087, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:07.987052 30527 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:28:07.987614 30527 webserver.cc:460] Webserver started at http://127.18.229.190:43229/ using document root <none> and password file <none>
I20260924 19:28:07.987859 30527 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:07.987896 30527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:07.988025 30527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:07 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278087, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:08.048910 30527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "a58ff495b4084d38858fc5fd32b177eb"
format_stamp: "Formatted at 2026-09-24 19:28:08 on dist-test-slave-r8cc"
server_key: "3195edd3ab93d303aead1ceef42855e0"
server_key_iv: "fb71301c9040130c3ac248ce9d78155d"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:08.049383 30527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "a58ff495b4084d38858fc5fd32b177eb"
format_stamp: "Formatted at 2026-09-24 19:28:08 on dist-test-slave-r8cc"
server_key: "3195edd3ab93d303aead1ceef42855e0"
server_key_iv: "fb71301c9040130c3ac248ce9d78155d"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:08.050782 30527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:28:08.064232 30542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:08.064469 30527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:28:08.064581 30527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "a58ff495b4084d38858fc5fd32b177eb"
format_stamp: "Formatted at 2026-09-24 19:28:08 on dist-test-slave-r8cc"
server_key: "3195edd3ab93d303aead1ceef42855e0"
server_key_iv: "fb71301c9040130c3ac248ce9d78155d"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:08.064683 30527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:08.072391 30527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:08.074395 30527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:08.074544 30527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:08.079265 30527 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:40817
I20260924 19:28:08.079300 30594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:40817 every 8 connection(s)
I20260924 19:28:08.079644 30527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:28:08.080435 30595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:08.083437 30595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb: Bootstrap starting.
I20260924 19:28:08.084141 30595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:08.084400 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 30527
I20260924 19:28:08.084476 30595 log.cc:824] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:08.084515 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:28:08.084728 19350 external_mini_cluster.cc:1428] Setting key 1bbfc7f981b9f929848736c4de027fca
I20260924 19:28:08.085240 30595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb: No bootstrap required, opened a new log
I20260924 19:28:08.087807 30595 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ff495b4084d38858fc5fd32b177eb" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40817 } }
I20260924 19:28:08.087992 30595 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:08.088027 30595 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58ff495b4084d38858fc5fd32b177eb, State: Initialized, Role: FOLLOWER
I20260924 19:28:08.088214 30595 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ff495b4084d38858fc5fd32b177eb" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40817 } }
I20260924 19:28:08.088302 30595 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:28:08.088331 30595 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:28:08.088395 30595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:08.088995 30595 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ff495b4084d38858fc5fd32b177eb" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40817 } }
I20260924 19:28:08.089120 30595 leader_election.cc:302] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58ff495b4084d38858fc5fd32b177eb; no voters:
I20260924 19:28:08.089273 30595 leader_election.cc:288] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:28:08.089299 30600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:28:08.089510 30600 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [term 1 LEADER]: Becoming Leader. State: Replica: a58ff495b4084d38858fc5fd32b177eb, State: Running, Role: LEADER
I20260924 19:28:08.089618 30595 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:28:08.089627 30600 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ff495b4084d38858fc5fd32b177eb" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40817 } }
I20260924 19:28:08.091234 30595 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:28:08.091321 30601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58ff495b4084d38858fc5fd32b177eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ff495b4084d38858fc5fd32b177eb" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40817 } } }
I20260924 19:28:08.091404 30601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:08.091693 30602 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58ff495b4084d38858fc5fd32b177eb. Latest consensus state: current_term: 1 leader_uuid: "a58ff495b4084d38858fc5fd32b177eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58ff495b4084d38858fc5fd32b177eb" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 40817 } } }
I20260924 19:28:08.091751 30602 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb [sys.catalog]: This master's current role is: LEADER
Sep 24 19:28:08 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:08.093832 30614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:28:08.095126 30614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:28:08.098552 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:08.085271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.18.229.190:40817)
0924 19:28:08.085444 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:08.085453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:08.085483 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:28:08.086450 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:08.086459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:08.086523 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:08.086674 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:08.087453 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:08.088865 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:08.089948 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:08.090680 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:08.090815 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:08.094112 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:08.094162 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:08.094166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:08.094284 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:08.096904 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:08.097361 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:08.097364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:08.097364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:08.097446 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:08.097655 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:08.097656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:08.097657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:08.097938 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:08.098118 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:08.098211 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 19:28:08.098340 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260924 19:28:08.100514 30614 catalog_manager.cc:1347] Generated new cluster ID: 36e62239072645fab6fdc0b5bb98729e
I20260924 19:28:08.100572 30614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:28:08.129930 30614 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:28:08.130733 30614 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:28:08.143033 30614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb: Generated new TSK 0
I20260924 19:28:08.143453 30614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:28:08.854943 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:45735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40817
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:08.994149 30640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:08.994375 30640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:08.994414 30640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:08.995579 30640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:08.995644 30640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:08.995688 30640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:28:08.997287 30640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.229.148:45735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:08.997582 30640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:08.997845 30640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:09.002904 30648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:09.003104 30645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:09.002993 30646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:09.003592 30640 server_base.cc:1034] running on GCE node
I20260924 19:28:09.003803 30640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:09.004052 30640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:09.005218 30640 hybrid_clock.cc:648] HybridClock initialized: now 1790278089005194 us; error 43 us; skew 500 ppm
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:09.007934 30640 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:28:09.008496 30640 webserver.cc:460] Webserver started at http://127.18.229.129:46577/ using document root <none> and password file <none>
I20260924 19:28:09.008757 30640 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:09.008841 30640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:09.008994 30640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:09.022429 30640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "b6e11bad479e42f1baf7cb34881af94d"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "0541fd7f9d6aff25e1e09c0a590abfd3"
server_key_iv: "bb10fc85cd0b5eba41b11a1c1a87fa1b"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.022991 30640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "b6e11bad479e42f1baf7cb34881af94d"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "0541fd7f9d6aff25e1e09c0a590abfd3"
server_key_iv: "bb10fc85cd0b5eba41b11a1c1a87fa1b"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.025590 30640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:28:09.039480 30655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.039709 30640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:09.039825 30640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "b6e11bad479e42f1baf7cb34881af94d"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "0541fd7f9d6aff25e1e09c0a590abfd3"
server_key_iv: "bb10fc85cd0b5eba41b11a1c1a87fa1b"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.039923 30640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:09.050940 30640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:09.053071 30640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:09.053238 30640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:09.053489 30640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:09.053858 30640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:09.053918 30640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.053959 30640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:09.053985 30640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.061214 30640 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:39307
I20260924 19:28:09.061369 30770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:39307 every 8 connection(s)
I20260924 19:28:09.061720 30640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:28:09.071273 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 30640
I20260924 19:28:09.071401 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:28:09.071619 19350 external_mini_cluster.cc:1428] Setting key 2f6bd755b740d50fcbcab620732095f9
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:09.076862 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.063610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:43839 (local address 127.18.229.190:40817)
0924 19:28:09.065649 (+ 2039us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.065652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.065661 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:28:09.065696 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.065697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.065710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.065777 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:09.066572 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.067388 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.069548 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.069876 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.069941 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.073129 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:09.073130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:09.073130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:09.073139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:09.075036 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.075579 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.075580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.075580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.075652 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.075998 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.075999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.075999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.076193 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:09.076273 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.076637 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.076734 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":75,"threads_started":1}
I20260924 19:28:09.076979 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.063324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40817 (local address 127.18.229.129:43839)
0924 19:28:09.064732 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.064778 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.065450 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.065910 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.065915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.066179 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:09.066449 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.066455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.067540 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.067541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.069432 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.069436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.070011 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.070012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.070152 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.071165 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:09.071176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:09.073010 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:09.075135 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:09.075150 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:09.075153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:09.075464 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:09.075736 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:09.075737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:09.075737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:09.075891 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:09.076263 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:09.076270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:09.076448 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.076564 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.076732 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":76,"threads_started":1}
I20260924 19:28:09.077606 30771 heartbeater.cc:347] Connected to a master server at 127.18.229.190:40817
I20260924 19:28:09.077715 30771 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:09.077920 30771 heartbeater.cc:511] Master 127.18.229.190:40817 requested a full tablet report, sending...
I20260924 19:28:09.078564 30559 ts_manager.cc:194] Registered new tserver with Master: b6e11bad479e42f1baf7cb34881af94d (127.18.229.129:39307)
I20260924 19:28:09.079576 30559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:43839
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:28:09.103075 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:45735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40817
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:09.213918 30777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:09.214210 30777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:09.214243 30777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:09.215912 30777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:09.215969 30777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:09.216008 30777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:28:09.218186 30777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.229.148:45735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:09.218441 30777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:09.218735 30777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:09.224802 30785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:09.224824 30787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:09.225201 30784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:09.225688 30777 server_base.cc:1034] running on GCE node
I20260924 19:28:09.225909 30777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:09.226190 30777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:09.227398 30777 hybrid_clock.cc:648] HybridClock initialized: now 1790278089227375 us; error 40 us; skew 500 ppm
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:09.230656 30777 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:28:09.231210 30777 webserver.cc:460] Webserver started at http://127.18.229.130:32877/ using document root <none> and password file <none>
I20260924 19:28:09.231456 30777 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:09.231531 30777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:09.231679 30777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:09.244252 30777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "cd595c8376d44228ad66ac9b2a2b1f45"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "801426d3154bc6c8edbde27057914546"
server_key_iv: "503fc0e8541b4740db4500808f352fd7"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.244884 30777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "cd595c8376d44228ad66ac9b2a2b1f45"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "801426d3154bc6c8edbde27057914546"
server_key_iv: "503fc0e8541b4740db4500808f352fd7"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.247121 30777 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260924 19:28:09.280668 30794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.280949 30777 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260924 19:28:09.281046 30777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "cd595c8376d44228ad66ac9b2a2b1f45"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "801426d3154bc6c8edbde27057914546"
server_key_iv: "503fc0e8541b4740db4500808f352fd7"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.281172 30777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:09.292740 30777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:09.294961 30777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:09.295131 30777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:09.295361 30777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:09.295727 30777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:09.295782 30777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.295848 30777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:09.295877 30777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.303131 30777 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:38109
I20260924 19:28:09.303260 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:38109 every 8 connection(s)
I20260924 19:28:09.303576 30777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:28:09.309527 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 30777
I20260924 19:28:09.309657 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:28:09.309880 19350 external_mini_cluster.cc:1428] Setting key aa3e0cf93f61ece2c797c85a7dbb6f6c
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:09.314280 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.304419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:42057 (local address 127.18.229.190:40817)
0924 19:28:09.304496 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.304498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.304770 (+ 272us) server_negotiation.cc:407] Connection header received
0924 19:28:09.305498 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.305500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.305519 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.305587 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:09.306351 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.307097 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.308284 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.308611 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.308690 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.311022 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:09.311023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:09.311024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:09.311032 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:09.312676 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.313238 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.313239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.313240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.313308 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.313603 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.313604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.313604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.313725 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:09.313797 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.314089 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.314189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:28:09.314505 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.304394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40817 (local address 127.18.229.130:42057)
0924 19:28:09.304606 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.304659 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.305369 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.305675 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.305679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.305930 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:09.306225 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.306232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.307212 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.307213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.308187 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.308190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.308702 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.308702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.308807 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.309409 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:09.309417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:09.310931 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:09.312777 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:09.312784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:09.312787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:09.313020 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:09.313383 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:09.313383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:09.313384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:09.313520 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:09.313800 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:09.313807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:09.313950 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.314055 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.314202 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260924 19:28:09.315068 30908 heartbeater.cc:347] Connected to a master server at 127.18.229.190:40817
I20260924 19:28:09.315191 30908 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:09.315371 30908 heartbeater.cc:511] Master 127.18.229.190:40817 requested a full tablet report, sending...
I20260924 19:28:09.315819 30559 ts_manager.cc:194] Registered new tserver with Master: cd595c8376d44228ad66ac9b2a2b1f45 (127.18.229.130:38109)
I20260924 19:28:09.316381 30559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:42057
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:28:09.338514 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:45735/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40817
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:09.459582 30914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:09.459846 30914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:09.459885 30914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:09.461614 30914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:09.461673 30914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:09.461710 30914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:28:09.463851 30914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.229.148:45735/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:09.464279 30914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:09.464552 30914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:09.474834 30919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:09.475220 30920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:09.475508 30922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:09.476586 30914 server_base.cc:1034] running on GCE node
I20260924 19:28:09.476807 30914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:09.477101 30914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:09.478294 30914 hybrid_clock.cc:648] HybridClock initialized: now 1790278089478264 us; error 44 us; skew 500 ppm
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:09.481745 30914 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:28:09.482239 30914 webserver.cc:460] Webserver started at http://127.18.229.131:35207/ using document root <none> and password file <none>
I20260924 19:28:09.482494 30914 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:09.482558 30914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:09.482695 30914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:09.502902 30914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "26fd7d3e60fb4a92b0233c0c84410601"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "218a41cffb2e3684e172343f6452f046"
server_key_iv: "925b0d6fa06e9db2fddca802e3139ed0"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.503360 30914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "26fd7d3e60fb4a92b0233c0c84410601"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "218a41cffb2e3684e172343f6452f046"
server_key_iv: "925b0d6fa06e9db2fddca802e3139ed0"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.505527 30914 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 19:28:09.520300 30929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.520654 30914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:28:09.520816 30914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "26fd7d3e60fb4a92b0233c0c84410601"
format_stamp: "Formatted at 2026-09-24 19:28:09 on dist-test-slave-r8cc"
server_key: "218a41cffb2e3684e172343f6452f046"
server_key_iv: "925b0d6fa06e9db2fddca802e3139ed0"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:09.520989 30914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:09.542081 30914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:09.544518 30914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:09.544785 30914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:09.545069 30914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:09.545440 30914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:09.545493 30914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.545531 30914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:09.545555 30914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:09.552659 30914 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:32957
I20260924 19:28:09.552775 31042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:32957 every 8 connection(s)
I20260924 19:28:09.553056 30914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:09.559798 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 30914
I20260924 19:28:09.559906 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:28:09.560137 19350 external_mini_cluster.cc:1428] Setting key 0ba06be5d1041caecb581e154e78da6c
I20260924 19:28:09.562716 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.554162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:52477 (local address 127.18.229.190:40817)
0924 19:28:09.554207 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.554210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.554467 (+ 257us) server_negotiation.cc:407] Connection header received
0924 19:28:09.554971 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.554972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.554985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.555042 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:09.555580 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.556214 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.557364 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.557570 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.557654 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.559778 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:09.559780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:09.559780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:09.559789 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:09.561393 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.561789 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.561790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.561790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.561847 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.562087 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.562088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.562088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.562203 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:09.562274 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.562549 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.562629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:28:09.563061 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.554167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:40817 (local address 127.18.229.131:52477)
0924 19:28:09.554353 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.554395 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.554868 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.555111 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.555115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.555303 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:09.555486 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.555490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.556305 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.556306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.557215 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.557218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.557664 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.557665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.557791 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.558308 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:09.558317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:09.559652 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:09.561489 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:09.561495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:09.561498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:09.561710 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:09.561917 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:09.561918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:09.561919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:09.562020 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:09.562285 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:09.562291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:09.562413 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.562512 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.562649 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":123,"threads_started":1}
I20260924 19:28:09.563839 31043 heartbeater.cc:347] Connected to a master server at 127.18.229.190:40817
I20260924 19:28:09.563936 31043 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:09.564296 31043 heartbeater.cc:511] Master 127.18.229.190:40817 requested a full tablet report, sending...
I20260924 19:28:09.564746 30559 ts_manager.cc:194] Registered new tserver with Master: 26fd7d3e60fb4a92b0233c0c84410601 (127.18.229.131:32957)
I20260924 19:28:09.565114 30559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:52477
I20260924 19:28:09.576076 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:28:09.585860 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.578405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.18.229.190:40817)
0924 19:28:09.578445 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.578447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.578568 (+ 121us) server_negotiation.cc:407] Connection header received
0924 19:28:09.578607 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.578608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.578617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.578688 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:09.579362 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.580026 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.581049 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.581354 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.581403 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.582854 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:09.582855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:09.582855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:09.582861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:09.584544 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.584958 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.584959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.584959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.585017 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:09.585284 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:09.585285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:09.585285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:09.585403 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:09.585472 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.585689 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.585770 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:28:09.587543 30559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:28:09.589890 30559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:28:09.597831 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.592499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:32957 (local address 127.0.0.1:33662)
0924 19:28:09.592662 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.592702 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.592766 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.593821 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.593825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.593827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:09.593891 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.593893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.595190 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.595191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.596507 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.596510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.597530 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.597531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.597615 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.597640 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.597712 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.597760 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260924 19:28:09.598150 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.592905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.18.229.131:32957)
0924 19:28:09.593621 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.593637 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.593646 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:28:09.593682 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.593684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.593713 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.593789 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:09.593974 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.594990 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.596649 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.597356 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.597442 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.597496 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.598034 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.598072 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":104,"threads_started":1}
I20260924 19:28:09.598817 30968 tablet_service.cc:1467] Processing CreateTablet for tablet 992ca7b9afee431987e7e7e579123a9a (DEFAULT_TABLE table=test-table [id=961d301acc7c4d4cb69e469d81d8435d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:28:09.599164 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992ca7b9afee431987e7e7e579123a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:09.602053 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.593100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38109 (local address 127.0.0.1:49854)
0924 19:28:09.593479 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.593487 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.593518 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.598582 (+ 5064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.598583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.598585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:09.598631 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.598633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.599889 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.599889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.600927 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.600929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.601870 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.601872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.601924 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.601930 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.601961 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.601993 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":69,"threads_started":1}
I20260924 19:28:09.602435 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.593213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.18.229.130:38109)
0924 19:28:09.598374 (+ 5161us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.598379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.598407 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:28:09.598448 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.598450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.598464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.598583 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:09.598711 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.599790 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.601059 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.601773 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.602224 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.602288 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.602314 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.602360 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5123,"thread_start_us":76,"threads_started":1}
I20260924 19:28:09.603013 30842 tablet_service.cc:1467] Processing CreateTablet for tablet 992ca7b9afee431987e7e7e579123a9a (DEFAULT_TABLE table=test-table [id=961d301acc7c4d4cb69e469d81d8435d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:28:09.603308 31061 tablet_bootstrap.cc:492] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Bootstrap starting.
I20260924 19:28:09.603340 30842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992ca7b9afee431987e7e7e579123a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:09.604413 31061 tablet_bootstrap.cc:654] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:09.604768 31061 log.cc:824] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:09.605574 31061 tablet_bootstrap.cc:492] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: No bootstrap required, opened a new log
I20260924 19:28:09.605639 31061 ts_tablet_manager.cc:1397] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:28:09.607587 31061 raft_consensus.cc:348] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.607739 31061 raft_consensus.cc:374] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:09.607774 31061 raft_consensus.cc:729] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fd7d3e60fb4a92b0233c0c84410601, State: Initialized, Role: FOLLOWER
I20260924 19:28:09.607880 31061 consensus_queue.cc:260] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.608112 31061 ts_tablet_manager.cc:1428] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:28:09.608351 31043 heartbeater.cc:503] Master 127.18.229.190:40817 was elected leader, sending a full tablet report...
I20260924 19:28:09.610250 31063 tablet_bootstrap.cc:492] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Bootstrap starting.
I20260924 19:28:09.610321 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.592897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39307 (local address 127.0.0.1:34480)
0924 19:28:09.593357 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.593366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.593406 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.606709 (+ 13303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.606710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.606713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:09.606750 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.606751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.608060 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.608061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.609114 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.609117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.610065 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.610066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.610127 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.610131 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.610212 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.610250 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":85,"threads_started":1}
I20260924 19:28:09.610697 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.592887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.18.229.129:39307)
0924 19:28:09.603293 (+ 10406us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.603298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.603309 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:28:09.603343 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.603345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.603373 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.603453 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:09.606905 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.607958 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.609213 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.609930 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.610041 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.610096 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.610577 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.610623 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10369,"thread_start_us":54,"threads_started":1}
I20260924 19:28:09.611361 31063 tablet_bootstrap.cc:654] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:09.611315 30703 tablet_service.cc:1467] Processing CreateTablet for tablet 992ca7b9afee431987e7e7e579123a9a (DEFAULT_TABLE table=test-table [id=961d301acc7c4d4cb69e469d81d8435d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:28:09.611618 30703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 992ca7b9afee431987e7e7e579123a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:09.611645 31063 log.cc:824] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:09.612371 31063 tablet_bootstrap.cc:492] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: No bootstrap required, opened a new log
I20260924 19:28:09.612432 31063 ts_tablet_manager.cc:1397] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:28:09.614351 31063 raft_consensus.cc:348] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.614493 31063 raft_consensus.cc:374] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:09.614527 31063 raft_consensus.cc:729] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd595c8376d44228ad66ac9b2a2b1f45, State: Initialized, Role: FOLLOWER
I20260924 19:28:09.614645 31063 consensus_queue.cc:260] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.614898 31063 ts_tablet_manager.cc:1428] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:28:09.615075 30908 heartbeater.cc:503] Master 127.18.229.190:40817 was elected leader, sending a full tablet report...
I20260924 19:28:09.615450 31066 tablet_bootstrap.cc:492] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d: Bootstrap starting.
I20260924 19:28:09.616189 31066 tablet_bootstrap.cc:654] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:09.616539 31066 log.cc:824] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:09.617273 31066 tablet_bootstrap.cc:492] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d: No bootstrap required, opened a new log
I20260924 19:28:09.617332 31066 ts_tablet_manager.cc:1397] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:28:09.619841 31066 raft_consensus.cc:348] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.619987 31066 raft_consensus.cc:374] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:09.620023 31066 raft_consensus.cc:729] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e11bad479e42f1baf7cb34881af94d, State: Initialized, Role: FOLLOWER
I20260924 19:28:09.620587 31066 consensus_queue.cc:260] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.620939 31066 ts_tablet_manager.cc:1428] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 19:28:09.620990 30771 heartbeater.cc:503] Master 127.18.229.190:40817 was elected leader, sending a full tablet report...
Sep 24 19:28:09 dist-test-slave-r8cc krb5kdc[30083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278089, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:09.701524 31069 raft_consensus.cc:482] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:28:09.701682 31069 raft_consensus.cc:504] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.702073 31069 leader_election.cc:288] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd595c8376d44228ad66ac9b2a2b1f45 (127.18.229.130:38109), 26fd7d3e60fb4a92b0233c0c84410601 (127.18.229.131:32957)
I20260924 19:28:09.706900 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.703124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:54493 (local address 127.18.229.131:32957)
0924 19:28:09.703165 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.703167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.703284 (+ 117us) server_negotiation.cc:407] Connection header received
0924 19:28:09.703379 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.703380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.703389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.703419 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:09.703650 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.704440 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.705954 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.706505 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.706558 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.706567 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.706793 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.706831 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:28:09.707299 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.703010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:32957 (local address 127.18.229.129:54493)
0924 19:28:09.703224 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.703233 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.703308 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.703482 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.703483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.703484 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:09.703565 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.703567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.704586 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.704587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.705847 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.705849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.706625 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.706626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.706703 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.706706 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.707175 (+ 469us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.707218 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260924 19:28:09.707890 30997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992ca7b9afee431987e7e7e579123a9a" candidate_uuid: "b6e11bad479e42f1baf7cb34881af94d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26fd7d3e60fb4a92b0233c0c84410601" is_pre_election: true
I20260924 19:28:09.708053 30997 raft_consensus.cc:2454] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6e11bad479e42f1baf7cb34881af94d in term 0.
I20260924 19:28:09.708328 30659 leader_election.cc:302] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fd7d3e60fb4a92b0233c0c84410601, b6e11bad479e42f1baf7cb34881af94d; no voters:
I20260924 19:28:09.708453 31069 raft_consensus.cc:2781] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:28:09.708494 31069 raft_consensus.cc:482] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:28:09.708516 31069 raft_consensus.cc:3037] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:09.709426 31069 raft_consensus.cc:504] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.709553 31069 leader_election.cc:288] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [CANDIDATE]: Term 1 election: Requested vote from peers cd595c8376d44228ad66ac9b2a2b1f45 (127.18.229.130:38109), 26fd7d3e60fb4a92b0233c0c84410601 (127.18.229.131:32957)
I20260924 19:28:09.709751 30997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992ca7b9afee431987e7e7e579123a9a" candidate_uuid: "b6e11bad479e42f1baf7cb34881af94d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26fd7d3e60fb4a92b0233c0c84410601"
I20260924 19:28:09.709820 30997 raft_consensus.cc:3037] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:09.710798 30997 raft_consensus.cc:2454] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6e11bad479e42f1baf7cb34881af94d in term 1.
I20260924 19:28:09.710958 30659 leader_election.cc:302] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fd7d3e60fb4a92b0233c0c84410601, b6e11bad479e42f1baf7cb34881af94d; no voters:
I20260924 19:28:09.711026 31069 raft_consensus.cc:2781] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:28:09.711138 31069 raft_consensus.cc:686] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [term 1 LEADER]: Becoming Leader. State: Replica: b6e11bad479e42f1baf7cb34881af94d, State: Running, Role: LEADER
I20260924 19:28:09.711215 31069 consensus_queue.cc:237] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } }
I20260924 19:28:09.712749 30559 catalog_manager.cc:5495] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d reported cstate change: term changed from 0 to 1, leader changed from <none> to b6e11bad479e42f1baf7cb34881af94d (127.18.229.129). New cstate: current_term: 1 leader_uuid: "b6e11bad479e42f1baf7cb34881af94d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6e11bad479e42f1baf7cb34881af94d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39307 } health_report { overall_health: HEALTHY } } }
I20260924 19:28:09.713467 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.702515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:48081 (local address 127.18.229.130:38109)
0924 19:28:09.702761 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.702764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.702771 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:28:09.702799 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.702800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.702811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.702858 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:09.710483 (+ 7625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.711188 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.712604 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.713078 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.713129 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.713138 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.713340 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.713399 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260924 19:28:09.715075 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.702288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38109 (local address 127.18.229.129:48081)
0924 19:28:09.702591 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.702603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:09.702662 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:09.710307 (+ 7645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:09.710308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:09.710311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:09.710365 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.710366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.711407 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.711408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.712509 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:09.712512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.713168 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.713168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:09.713251 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.713255 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:28:09.714682 (+ 1427us) client_negotiation.cc:240] Negotiation successful
0924 19:28:09.714977 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":95,"threads_started":1}
I20260924 19:28:09.716142 30846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992ca7b9afee431987e7e7e579123a9a" candidate_uuid: "b6e11bad479e42f1baf7cb34881af94d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" is_pre_election: true
I20260924 19:28:09.716152 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "992ca7b9afee431987e7e7e579123a9a" candidate_uuid: "b6e11bad479e42f1baf7cb34881af94d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd595c8376d44228ad66ac9b2a2b1f45"
I20260924 19:28:09.716307 30846 raft_consensus.cc:2454] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6e11bad479e42f1baf7cb34881af94d in term 0.
I20260924 19:28:09.719157 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.716239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.18.229.129:39307)
0924 19:28:09.716307 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:09.716309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:09.716316 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:28:09.716345 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:09.716346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:09.716363 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:09.716422 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:28:09.716633 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.717331 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.718076 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:09.718365 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:09.718428 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:09.718622 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:28:09.718753 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:28:09.718823 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:09.719052 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 19:28:09.719072 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:28:09.721190 30847 raft_consensus.cc:3037] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:09.721237 30997 raft_consensus.cc:1261] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 1 FOLLOWER]: Refusing update from remote peer b6e11bad479e42f1baf7cb34881af94d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:28:09.721537 31072 consensus_queue.cc:1035] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26fd7d3e60fb4a92b0233c0c84410601" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:28:09.722172 30847 raft_consensus.cc:1261] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 1 FOLLOWER]: Refusing update from remote peer b6e11bad479e42f1baf7cb34881af94d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:28:09.723026 31072 consensus_queue.cc:1035] T 992ca7b9afee431987e7e7e579123a9a P b6e11bad479e42f1baf7cb34881af94d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd595c8376d44228ad66ac9b2a2b1f45" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:28:09.725610 31075 mvcc.cc:205] Tried to move back new op lower bound from 7332979055495159808 to 7332979055457873920. Current Snapshot: MvccSnapshot[applied={T|T < 7332979055495159808 or (T in {7332979055495159808})}]
I20260924 19:28:09.727416 30558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:28:09.727504 30558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:28:09.728405 30558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a58ff495b4084d38858fc5fd32b177eb: Sending DeleteTablet for 3 replicas of tablet 992ca7b9afee431987e7e7e579123a9a
I20260924 19:28:09.728761 30703 tablet_service.cc:1514] Processing DeleteTablet for tablet 992ca7b9afee431987e7e7e579123a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:09 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:34480
I20260924 19:28:09.728969 30968 tablet_service.cc:1514] Processing DeleteTablet for tablet 992ca7b9afee431987e7e7e579123a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:09 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:33662
I20260924 19:28:09.729413 30842 tablet_service.cc:1514] Processing DeleteTablet for tablet 992ca7b9afee431987e7e7e579123a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:09 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:49854
I20260924 19:28:09.729956 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 30640
I20260924 19:28:09.734157 31085 tablet_replica.cc:330] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: stopping tablet replica
I20260924 19:28:09.734297 31085 raft_consensus.cc:2229] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:28:09.734364 31085 pending_rounds.cc:62] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Trying to abort 2 pending ops.
I20260924 19:28:09.734571 31085 pending_rounds.cc:68] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332979055495159808 op_type: WRITE_OP write_request { tablet_id: "992ca7b9afee431987e7e7e579123a9a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\365\365\325\325\006\"8\n\ntest-admin\022*\n 961d301acc7c4d4cb69e469d81d8435d\020\001\030\001 \001(\001" signature: "<redacted>""\'\330_\375B\206\303\340\255\240eb\345\247(E\220me\325\354\206\035%\n\000\321P\306Q\037W`\262\227\271\371\250\363\2547\324\260\243e8\305\031\307vT>:\023\252#C\374\276a\0207\375{" signing_key_seq_num: 0 } } request_id { client_id: "ff3ca4ecaf7146deb4eb63c169efb845" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 19:28:09.734803 31085 pending_rounds.cc:68] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332979055457873920 op_type: NO_OP noop_request { }
I20260924 19:28:09.734860 31085 raft_consensus.cc:2866] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:28:09.734894 31085 raft_consensus.cc:2258] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:28:09.738888 31084 tablet_replica.cc:330] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: stopping tablet replica
I20260924 19:28:09.739001 31084 raft_consensus.cc:2229] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:28:09.739116 31084 raft_consensus.cc:2258] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:28:09.739537 31085 ts_tablet_manager.cc:1905] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:28:09.741214 30795 connection.cc:535] server connection from 127.18.229.129:48081 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:28:09.741540 31084 ts_tablet_manager.cc:1905] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:28:09.742986 31084 ts_tablet_manager.cc:1918] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:28:09.743057 31084 log.cc:1196] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/992ca7b9afee431987e7e7e579123a9a
I20260924 19:28:09.743316 31084 ts_tablet_manager.cc:1939] T 992ca7b9afee431987e7e7e579123a9a P 26fd7d3e60fb4a92b0233c0c84410601: Deleting consensus metadata
I20260924 19:28:09.743496 31085 ts_tablet_manager.cc:1918] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:28:09.743554 31085 log.cc:1196] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/992ca7b9afee431987e7e7e579123a9a
I20260924 19:28:09.744153 31085 ts_tablet_manager.cc:1939] T 992ca7b9afee431987e7e7e579123a9a P cd595c8376d44228ad66ac9b2a2b1f45: Deleting consensus metadata
I20260924 19:28:09.744621 30543 catalog_manager.cc:4841] TS cd595c8376d44228ad66ac9b2a2b1f45 (127.18.229.130:38109): tablet 992ca7b9afee431987e7e7e579123a9a (table test-table [id=961d301acc7c4d4cb69e469d81d8435d]) successfully deleted
I20260924 19:28:09.745142 30546 catalog_manager.cc:4841] TS 26fd7d3e60fb4a92b0233c0c84410601 (127.18.229.131:32957): tablet 992ca7b9afee431987e7e7e579123a9a (table test-table [id=961d301acc7c4d4cb69e469d81d8435d]) successfully deleted
W20260924 19:28:09.745445 30545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:39307 (error 108)
I20260924 19:28:09.746332 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:09.746226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39307 (local address 127.0.0.1:34498)
0924 19:28:09.746270 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:09.746300 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:28:09.746515 30545 catalog_manager.cc:4584] TS b6e11bad479e42f1baf7cb34881af94d (127.18.229.129:39307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 992ca7b9afee431987e7e7e579123a9a: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39307: connect: Connection refused (error 111)
I20260924 19:28:09.746748 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 30777
I20260924 19:28:09.751784 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 30914
I20260924 19:28:09.759302 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 30527
I20260924 19:28:09.766857 19350 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:28:10.111714 19350 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:28:10.111816 19350 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:28:10.457029 19350 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:28:10.457 UTC [30113] LOG: received smart shutdown request
2026-09-24 19:28:10.460 UTC [30113] LOG: background worker "logical replication launcher" (PID 30122) exited with exit code 1
2026-09-24 19:28:10.460 UTC [30117] LOG: shutting down
2026-09-24 19:28:10.616 UTC [30113] LOG: database system is shut down
2026-09-24T19:28:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45230 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:28:10 dist-test-slave-r8cc krb5kdc[31095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:28:10 dist-test-slave-r8cc krb5kdc[31095](info): set up 2 sockets
Sep 24 19:28:10 dist-test-slave-r8cc krb5kdc[31095](info): commencing operation
krb5kdc: starting...
W20260924 19:28:11.748279 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:28:11 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:28:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/rangeradmin_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/rangeradmin_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/rangerlookup_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/rangerlookup_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
I20260924 19:28:11.914150 19350 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2KIbLR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/postgres -l logfile start
2026-09-24 19:28:14.303 UTC [31125] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:28:14.303 UTC [31125] LOG: listening on IPv4 address "127.18.229.148", port 35569
2026-09-24 19:28:14.307 UTC [31125] LOG: listening on Unix socket "/tmp/.s.PGSQL.35569"
2026-09-24 19:28:14.329 UTC [31128] LOG: database system was shut down at 2026-09-24 19:28:12 UTC
2026-09-24 19:28:14.332 UTC [31125] LOG: database system is ready to accept connections
I20260924 19:28:15.283910 19350 mini_postgres.cc:96] Postgres bound to 35569
2026-09-24 19:28:15.289 UTC [31136] FATAL: database "slave" does not exist
127.18.229.148:35569 - accepting connections
I20260924 19:28:15.290266 19350 mini_ranger.cc:161] Starting Ranger...
I20260924 19:28:15.304525 19350 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:28:15.761307 19350 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:28:15.761370 19350 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-admin
2026-09-24 19:28:16,034 [I] DB FLAVOR :POSTGRES
2026-09-24 19:28:16,034 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:28:16,035 [I] Checking connection..
2026-09-24 19:28:16,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:28:16,272 [I] Checking connection passed.
2026-09-24 19:28:16,272 [I] --------- Verifying version history table ---------
2026-09-24 19:28:16,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:28:16,531 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:28:16,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:28:16,754 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:28:16,755 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:28:16,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:28:16.994 UTC [31200] WARNING: there is no transaction in progress
2026-09-24 19:28:17,000 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:28:17,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:28:17,224 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:28:17,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:28:17,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:28:17,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:28:17,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r8cc.c.gcp-upstream.internal','N') ;"
2026-09-24 19:28:17,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:28:18,223 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:28:18,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:28:18,448 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:28:18,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:28:18,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:28:18,922 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:28:18,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:28:19.881 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:19.897 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:19.913 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:19.929 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:19.945 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:19.961 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.018 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.025 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.038 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.048 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.058 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.072 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.081 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.088 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.095 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20.101 UTC [31308] WARNING: there is no transaction in progress
2026-09-24 19:28:20,718 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:28:20,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:28:20,958 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:28:20,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:28:21,188 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:28:21,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:28:21,443 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:28:21,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:28:21,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:28:21,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r8cc.c.gcp-upstream.internal';"
2026-09-24 19:28:22,173 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:28:22,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:28:22,429 [I] DB_PATCHES have already been applied
I20260924 19:28:22.434818 19350 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-admin:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2KIbLR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:28:22.434916 19350 mini_ranger.cc:193] Using host: 127.18.229.148
I20260924 19:28:22.437989 19350 mini_ranger.cc:227] Ranger admin URL: http://127.18.229.148:52361
Sep 24, 2026 7:28:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:28:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:28:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:28:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:28:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:28:23.550369 19350 mini_ranger.cc:160] Time spent starting Ranger: real 8.260s user 0.001s sys 0.008s
Sep 24, 2026 7:28:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:28:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:28:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:28:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:28:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:28:43.811837 19350 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.229.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.229.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/rangerkms_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.229.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/rangerkms_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.229.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.229.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.229.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/HTTP_127.18.229.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:28:43 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278123, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:28:43.897454 19350 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:28:43.906808 19350 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:28:44.776481 19350 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:28:44.776584 19350 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms
I20260924 19:28:44.776605 19350 mini_ranger_kms.cc:227] Using postgres at 127.18.229.148:35569
2026-09-24 19:28:44,936 [I] DB FLAVOR :POSTGRES
2026-09-24 19:28:44,937 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:28:44,937 [I] Checking connection
2026-09-24 19:28:44,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:28:45,192 [I] connection success
2026-09-24 19:28:45,192 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:28:45,192 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:28:45,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:28:45,424 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:28:45,424 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:28:45,424 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:28:45,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.229.148:35569/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:28:45,684 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:28:46.132925 19350 mini_ranger_kms.cc:310] Created kms service
I20260924 19:28:46.325865 19350 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:28:46.368269 19350 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:28:46 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278123, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:46.736413 19350 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:28:46.736656 19350 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms:/tmp/dist-test-task2KIbLR/build/release/bin/postgresql.jar:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2KIbLR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:28:46.736686 19350 mini_ranger_kms.cc:240] Using host: 127.18.229.148
I20260924 19:28:46.740481 19350 mini_ranger_kms.cc:276] Ranger KMS PID: 31488
I20260924 19:28:46.740562 19350 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.229.148:37797
Sep 24, 2026 7:28:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:28:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:28:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:28:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:28:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:28:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:28:48.931669 19350 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.034s user 0.000s sys 0.143s
I20260924 19:28:48.931759 19350 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:28:48.931804 19350 mini_ranger_kms.cc:401] 127.18.229.148:37797/kms/v1/keys
Sep 24 19:28:49 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278129, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.229.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:28:49 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278129, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.229.148@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
Sep 24 19:28:49 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278129, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.229.148@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
Sep 24 19:28:52 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:28:52.115784 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43225
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43225
--ranger_config_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:52.211160 31539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:52.211369 31539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:52.211390 31539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:52.212668 31539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:28:52.212711 31539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:52.212728 31539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:52.212739 31539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:28:52.212788 31539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:28:52.214349 31539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.229.148:37797/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43225
--ranger_config_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:52.214610 31539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:52.214838 31539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:52.219852 31545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:52.219897 31539 server_base.cc:1034] running on GCE node
W20260924 19:28:52.219863 31544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:52.220031 31547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:52.220353 31539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:52.220712 31539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:52.221875 31539 hybrid_clock.cc:648] HybridClock initialized: now 1790278132221842 us; error 47 us; skew 500 ppm
Sep 24 19:28:52 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:52.224828 31539 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:28:52.225311 31539 webserver.cc:460] Webserver started at http://127.18.229.190:32983/ using document root <none> and password file <none>
I20260924 19:28:52.225543 31539 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:52.225610 31539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:52.225765 31539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:52 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:52.288084 31539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "d3be9aace2934d74b4486051710c3399"
format_stamp: "Formatted at 2026-09-24 19:28:52 on dist-test-slave-r8cc"
server_key: "544d37881cbaccfcfc1ee89de87c5934"
server_key_iv: "d621928d2d556674f18ac5dd61867257"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:52.288589 31539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "d3be9aace2934d74b4486051710c3399"
format_stamp: "Formatted at 2026-09-24 19:28:52 on dist-test-slave-r8cc"
server_key: "544d37881cbaccfcfc1ee89de87c5934"
server_key_iv: "d621928d2d556674f18ac5dd61867257"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:52.290387 31539 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 19:28:52.305521 31554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:52.305819 31539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:52.305927 31539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "d3be9aace2934d74b4486051710c3399"
format_stamp: "Formatted at 2026-09-24 19:28:52 on dist-test-slave-r8cc"
server_key: "544d37881cbaccfcfc1ee89de87c5934"
server_key_iv: "d621928d2d556674f18ac5dd61867257"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:52.306022 31539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:52.324083 31539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:52.326277 31539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:52.326431 31539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:52.331427 31539 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:43225
I20260924 19:28:52.331521 31606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:43225 every 8 connection(s)
I20260924 19:28:52.331882 31539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:28:52.332357 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 31539
I20260924 19:28:52.332469 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:28:52.332695 19350 external_mini_cluster.cc:1428] Setting key 7e671da23690e6d6d634c2b7c256731e
I20260924 19:28:52.332753 31607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:52.336097 31607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Bootstrap starting.
I20260924 19:28:52.336771 31607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:52.337060 31607 log.cc:824] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:52.337883 31607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: No bootstrap required, opened a new log
Sep 24 19:28:52 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:52.340395 31607 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:52.340677 31607 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:52.340734 31607 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3be9aace2934d74b4486051710c3399, State: Initialized, Role: FOLLOWER
I20260924 19:28:52.340867 31607 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:52.340973 31607 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:28:52.341023 31607 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:28:52.341073 31607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:52.341773 31607 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:52.341877 31607 leader_election.cc:302] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3be9aace2934d74b4486051710c3399; no voters:
I20260924 19:28:52.342088 31607 leader_election.cc:288] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:28:52.342144 31612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:28:52.342355 31612 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 LEADER]: Becoming Leader. State: Replica: d3be9aace2934d74b4486051710c3399, State: Running, Role: LEADER
I20260924 19:28:52.342411 31607 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:28:52.342533 31612 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:52.342912 31612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3be9aace2934d74b4486051710c3399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } } }
I20260924 19:28:52.343225 31612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:52.343402 31612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3be9aace2934d74b4486051710c3399. Latest consensus state: current_term: 1 leader_uuid: "d3be9aace2934d74b4486051710c3399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } } }
I20260924 19:28:52.343538 31612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:52.343717 31621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:28:52.343853 31607 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:28:52.344302 31621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:28:52.349774 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:52.333238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.18.229.190:43225)
0924 19:28:52.333427 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:52.333436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:52.333467 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:28:52.333943 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:52.333953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:52.334017 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:52.334179 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:52.334958 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:52.336335 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:52.337337 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:52.337818 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:52.337970 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:52.340373 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:52.340377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:52.340377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:52.340396 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:52.342338 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:52.342785 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:52.342788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:52.342788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:52.342887 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:52.343106 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:52.343106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:52.343107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:52.343415 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:52.344037 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:52.344142 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:28:52.344264 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260924 19:28:52.351655 31621 catalog_manager.cc:1347] Generated new cluster ID: d90b71cf6ae940d681e9acdb0ec5082e
I20260924 19:28:52.351707 31621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:28:52.359721 31621 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:28:52.360766 31621 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:28:52.371053 31621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Generated new TSK 0
I20260924 19:28:52.371335 31621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:28:53.042997 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43225
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:53.150050 31652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:53.150322 31652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:53.150357 31652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:53.152098 31652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:53.152225 31652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:53.152303 31652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:28:53.154505 31652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.229.148:37797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:53.154757 31652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:53.155055 31652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:53.161381 31658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.161397 31657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.161672 31660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:53.162158 31652 server_base.cc:1034] running on GCE node
I20260924 19:28:53.162370 31652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:53.162642 31652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:53.163801 31652 hybrid_clock.cc:648] HybridClock initialized: now 1790278133163770 us; error 47 us; skew 500 ppm
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:53.166603 31652 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:28:53.167143 31652 webserver.cc:460] Webserver started at http://127.18.229.129:33629/ using document root <none> and password file <none>
I20260924 19:28:53.167399 31652 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:53.167469 31652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:53.167613 31652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:53.182257 31652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "dbdb476103b84f4c9fb3dc1873bd5435"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "76cca43fc5157c55d02d98bb3da797e1"
server_key_iv: "2b433d5864550767747da7c42063b1c2"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.182689 31652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "dbdb476103b84f4c9fb3dc1873bd5435"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "76cca43fc5157c55d02d98bb3da797e1"
server_key_iv: "2b433d5864550767747da7c42063b1c2"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.196540 31652 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.002s sys 0.000s
I20260924 19:28:53.212281 31669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.212522 31652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:28:53.212622 31652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "dbdb476103b84f4c9fb3dc1873bd5435"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "76cca43fc5157c55d02d98bb3da797e1"
server_key_iv: "2b433d5864550767747da7c42063b1c2"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.212713 31652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:53.232045 31652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:53.234606 31652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:53.234812 31652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:53.235098 31652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:53.235597 31652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:53.235648 31652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.235677 31652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:53.235694 31652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.243064 31652 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:44675
I20260924 19:28:53.243495 31652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:28:53.251103 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 31652
I20260924 19:28:53.251206 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:28:53.251376 19350 external_mini_cluster.cc:1428] Setting key 5ce68e15ef3f567ffa07b291178dbdcb
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:53.256798 31786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:53.244319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43225 (local address 127.18.229.129:43463)
0924 19:28:53.244673 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:53.244716 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:53.245387 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:53.245730 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:53.245735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:53.246029 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:53.246308 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:53.246314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.247372 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.247373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:53.249889 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:53.249892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.250456 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.250457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:53.250596 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:53.251232 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:53.251242 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:53.252960 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:53.255077 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:53.255084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:53.255087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:53.255389 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:53.255609 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:53.255610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:53.255610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:53.255744 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:53.256049 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:53.256056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:53.256248 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 19:28:53.256376 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 19:28:53.256546 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":98,"threads_started":1}
I20260924 19:28:53.257365 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:53.244324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:43463 (local address 127.18.229.190:43225)
0924 19:28:53.244612 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:53.244614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:53.244832 (+ 218us) server_negotiation.cc:407] Connection header received
0924 19:28:53.245522 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:53.245524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:53.245539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:53.245633 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:53.246426 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.247262 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.250003 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.250313 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.250375 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:53.253082 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:53.253083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:53.253084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:53.253091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:53.254969 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:53.255473 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:53.255474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:53.255475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:53.255542 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:53.255813 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:53.255814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:53.255814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:53.255954 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:53.256024 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:53.257182 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0924 19:28:53.257268 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":79,"threads_started":1}
I20260924 19:28:53.258133 31783 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43225
I20260924 19:28:53.258236 31783 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:53.258424 31783 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:53.259032 31571 ts_manager.cc:194] Registered new tserver with Master: dbdb476103b84f4c9fb3dc1873bd5435 (127.18.229.129:44675)
I20260924 19:28:53.259919 31571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:43463
I20260924 19:28:53.244506 31782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:44675 every 8 connection(s)
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:28:53.295527 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43225
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:28:53.414538 31790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:53.414832 31790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:53.414868 31790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:53.416639 31790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:53.416769 31790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:53.416859 31790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:28:53.419111 31790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.229.148:37797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:53.419590 31790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:53.419905 31790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:53.425072 31799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.425890 31797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.426080 31796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:53.426293 31790 server_base.cc:1034] running on GCE node
I20260924 19:28:53.426538 31790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:53.426823 31790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:53.427999 31790 hybrid_clock.cc:648] HybridClock initialized: now 1790278133427978 us; error 35 us; skew 500 ppm
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:53.430678 31790 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:28:53.431198 31790 webserver.cc:460] Webserver started at http://127.18.229.130:36269/ using document root <none> and password file <none>
I20260924 19:28:53.431411 31790 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:53.431452 31790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:53.431543 31790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:53.444438 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "04af23abdd7844ecab84ad8e49a9f9c4"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "6ce3c5973c062cb7151fad456266ae90"
server_key_iv: "92a6d1d2bd4fd57463335850304e461a"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.445029 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "04af23abdd7844ecab84ad8e49a9f9c4"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "6ce3c5973c062cb7151fad456266ae90"
server_key_iv: "92a6d1d2bd4fd57463335850304e461a"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.446812 31790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:53.460026 31806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.460319 31790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:53.460413 31790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "04af23abdd7844ecab84ad8e49a9f9c4"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "6ce3c5973c062cb7151fad456266ae90"
server_key_iv: "92a6d1d2bd4fd57463335850304e461a"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.460508 31790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:53.482659 31790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:53.485183 31790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:53.485389 31790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:53.485679 31790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:53.486093 31790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:53.486186 31790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.486261 31790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:53.486328 31790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.492980 31790 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:38537
I20260924 19:28:53.493467 31790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:28:53.500929 31920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:38537 every 8 connection(s)
I20260924 19:28:53.501747 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 31790
I20260924 19:28:53.501865 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:28:53.502067 19350 external_mini_cluster.cc:1428] Setting key 46c9efbd162c069d3f35876f484c84ba
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:53.522655 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:53.502038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:58573 (local address 127.18.229.190:43225)
0924 19:28:53.502099 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:53.502102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:53.510411 (+ 8309us) server_negotiation.cc:407] Connection header received
0924 19:28:53.511283 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:53.511286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:53.511306 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:53.511342 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:53.512315 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.513099 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.514661 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.514965 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.515014 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:53.519166 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:53.519167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:53.519168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:53.519177 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:53.520957 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:53.521462 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:53.521463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:53.521463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:53.521523 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:53.521833 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:53.521834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:53.521835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:53.521976 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:53.522031 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:53.522448 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 19:28:53.522542 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:28:53.523475 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:53.501732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43225 (local address 127.18.229.130:58573)
0924 19:28:53.510237 (+ 8505us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:53.510304 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:53.511099 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:53.511566 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:53.511571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:53.511819 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:53.512188 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:53.512196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.513217 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.513218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:53.514537 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:53.514541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.515161 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.515162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:53.516760 (+ 1598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:53.517394 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:53.517405 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:53.519049 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:53.521055 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:53.521062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:53.521065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:53.521366 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:53.521605 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:53.521605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:53.521606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:53.521748 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:53.522067 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:53.522074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:53.522265 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 19:28:53.523016 (+ 751us) client_negotiation.cc:240] Negotiation successful
0924 19:28:53.523198 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8443,"thread_start_us":90,"threads_started":1}
I20260924 19:28:53.524248 31921 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43225
I20260924 19:28:53.524356 31921 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:53.524585 31921 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:53.525004 31571 ts_manager.cc:194] Registered new tserver with Master: 04af23abdd7844ecab84ad8e49a9f9c4 (127.18.229.130:38537)
I20260924 19:28:53.525458 31571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:58573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:28:53.552752 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43225
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:53.672534 31926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:53.672902 31926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:53.672992 31926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:53.674677 31926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:53.674785 31926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:53.674866 31926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:28:53.676985 31926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.229.148:37797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:53.677439 31926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:53.677771 31926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:53.684780 31932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:53.685086 31926 server_base.cc:1034] running on GCE node
W20260924 19:28:53.685230 31934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.685243 31931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:53.685604 31926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:53.685861 31926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:53.687036 31926 hybrid_clock.cc:648] HybridClock initialized: now 1790278133687013 us; error 33 us; skew 500 ppm
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:53.690338 31926 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:28:53.690845 31926 webserver.cc:460] Webserver started at http://127.18.229.131:44007/ using document root <none> and password file <none>
I20260924 19:28:53.691089 31926 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:53.691138 31926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:53.691258 31926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:53.714854 31926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "ed71fb9161574bff924631db1fc9b4bc"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "dfb276bb74b7b01d84279ab4d161c050"
server_key_iv: "dbf48cc18ed83ae78c70d4431e8359ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.715296 31926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "ed71fb9161574bff924631db1fc9b4bc"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "dfb276bb74b7b01d84279ab4d161c050"
server_key_iv: "dbf48cc18ed83ae78c70d4431e8359ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.721894 31926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.002s
I20260924 19:28:53.736068 31941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.736421 31926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:28:53.736545 31926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "ed71fb9161574bff924631db1fc9b4bc"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "dfb276bb74b7b01d84279ab4d161c050"
server_key_iv: "dbf48cc18ed83ae78c70d4431e8359ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.736650 31926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:53.750389 31926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:53.753257 31926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:53.753441 31926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:53.753762 31926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:53.754135 31926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:53.754184 31926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.754216 31926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:53.754290 31926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.766330 31926 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:45865
I20260924 19:28:53.766914 31926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:28:53.769930 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 31926
I20260924 19:28:53.770043 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:28:53.770226 19350 external_mini_cluster.cc:1428] Setting key f5985c915e9d9a37ae0db09efb4bea7a
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:53.768427 32054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:45865 every 8 connection(s)
I20260924 19:28:53.781584 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:53.769494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:36425 (local address 127.18.229.190:43225)
0924 19:28:53.769545 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:53.769548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:53.769796 (+ 248us) server_negotiation.cc:407] Connection header received
0924 19:28:53.770784 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:53.770786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:53.770800 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:53.770835 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:53.771798 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.772746 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.774450 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.774780 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.774829 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:53.777723 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:53.777726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:53.777727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:53.777736 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:53.779661 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:53.780238 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:53.780239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:53.780239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:53.780304 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:53.780751 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:53.780752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:53.780752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:53.780907 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:53.780969 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:53.781395 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 19:28:53.781489 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:28:53.782420 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:53.769191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43225 (local address 127.18.229.131:36425)
0924 19:28:53.769653 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:53.769704 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:53.770644 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:53.771064 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:53.771068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:53.771334 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:53.771639 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:53.771647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.772878 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.772879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:53.774337 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:53.774340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:53.774939 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:53.774940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:53.775099 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:53.775721 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:53.775741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:53.777601 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:53.779769 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:53.779776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:53.779779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:53.780139 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:53.780519 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:53.780520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:53.780521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:53.780665 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:53.781023 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:53.781030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:53.781213 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 19:28:53.781975 (+ 762us) client_negotiation.cc:240] Negotiation successful
0924 19:28:53.782163 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":83,"threads_started":1}
I20260924 19:28:53.783152 32055 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43225
I20260924 19:28:53.783252 32055 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:53.783458 32055 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:53.783907 31571 ts_manager.cc:194] Registered new tserver with Master: ed71fb9161574bff924631db1fc9b4bc (127.18.229.131:45865)
I20260924 19:28:53.784502 31571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:36425
I20260924 19:28:53.792600 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:28:53.793331 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 31652
I20260924 19:28:53.801098 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 31790
I20260924 19:28:53.807022 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 31926
I20260924 19:28:53.813145 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 31539
I20260924 19:28:53.820472 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43225
--webserver_interface=127.18.229.190
--webserver_port=32983
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43225
--ranger_config_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:53.908826 32058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:53.909063 32058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:53.909099 32058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:53.910260 32058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:28:53.910331 32058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:53.910353 32058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:53.910372 32058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:28:53.910399 32058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:28:53.912055 32058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43225
--ranger_config_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=32983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:53.912358 32058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:53.912586 32058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:53.918422 32063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.918413 32064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:53.918428 32066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:53.918645 32058 server_base.cc:1034] running on GCE node
I20260924 19:28:53.918941 32058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:53.919252 32058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:53.920413 32058 hybrid_clock.cc:648] HybridClock initialized: now 1790278133920386 us; error 43 us; skew 500 ppm
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:53.922878 32058 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:28:53.923382 32058 webserver.cc:460] Webserver started at http://127.18.229.190:32983/ using document root <none> and password file <none>
I20260924 19:28:53.923629 32058 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:53.923679 32058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:28:53 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278133, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:53.961910 32058 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:53.962682 32074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:53.962843 32058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:53.962956 32058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "d3be9aace2934d74b4486051710c3399"
format_stamp: "Formatted at 2026-09-24 19:28:52 on dist-test-slave-r8cc"
server_key: "544d37881cbaccfcfc1ee89de87c5934"
server_key_iv: "d621928d2d556674f18ac5dd61867257"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:53.963327 32058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:53.971788 32058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:53.973491 32058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:53.973647 32058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:53.977873 32058 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:43225
I20260924 19:28:53.977921 32126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:43225 every 8 connection(s)
I20260924 19:28:53.978291 32058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:28:53.979427 32127 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:28:53.980079 32127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Bootstrap starting.
I20260924 19:28:53.987735 32127 log.cc:824] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:53.988305 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 32058
I20260924 19:28:53.988624 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:44675
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:43225
--webserver_port=33629
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:28:53.989976 32127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:28:53.990254 32127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Bootstrap complete.
I20260924 19:28:53.992791 32127 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:53.993013 32127 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3be9aace2934d74b4486051710c3399, State: Initialized, Role: FOLLOWER
I20260924 19:28:53.993180 32127 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:53.993251 32127 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:28:53.993299 32127 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:28:53.993346 32127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:28:53.994295 32127 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:53.994397 32127 leader_election.cc:302] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3be9aace2934d74b4486051710c3399; no voters:
I20260924 19:28:53.994596 32127 leader_election.cc:288] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:28:53.994640 32131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:28:53.994779 32131 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [term 2 LEADER]: Becoming Leader. State: Replica: d3be9aace2934d74b4486051710c3399, State: Running, Role: LEADER
I20260924 19:28:53.994907 32127 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:28:53.994922 32131 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } }
I20260924 19:28:53.995307 32133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3be9aace2934d74b4486051710c3399. Latest consensus state: current_term: 2 leader_uuid: "d3be9aace2934d74b4486051710c3399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } } }
I20260924 19:28:53.995385 32133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:53.995731 32132 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d3be9aace2934d74b4486051710c3399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3be9aace2934d74b4486051710c3399" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43225 } } }
I20260924 19:28:53.995800 32132 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:53.995929 32137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:28:53.996563 32137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:28:53.996731 32137 catalog_manager.cc:1259] Loaded cluster ID: d90b71cf6ae940d681e9acdb0ec5082e
I20260924 19:28:53.996800 32137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:28:53.996922 32127 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:28:53.997972 32137 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:28:53.998150 32137 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Loaded TSK: 0
I20260924 19:28:53.998361 32137 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:28:54.102226 32130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:54.102555 32130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:54.102651 32130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:54.104403 32130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:54.104519 32130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:54.104602 32130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:28:54.106645 32130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:44675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=33629
--tserver_master_addrs=127.18.229.190:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:54.106886 32130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:54.107196 32130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:54.113417 32165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:54.113755 32167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:54.114020 32130 server_base.cc:1034] running on GCE node
W20260924 19:28:54.114080 32164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:54.114264 32130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:54.114543 32130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:54.115716 32130 hybrid_clock.cc:648] HybridClock initialized: now 1790278134115685 us; error 43 us; skew 500 ppm
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:54.118780 32130 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:28:54.119311 32130 webserver.cc:460] Webserver started at http://127.18.229.129:33629/ using document root <none> and password file <none>
I20260924 19:28:54.119580 32130 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:54.119661 32130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:54.147970 32130 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 19:28:54.149226 32175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.149485 32130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:28:54.149580 32130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "dbdb476103b84f4c9fb3dc1873bd5435"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "76cca43fc5157c55d02d98bb3da797e1"
server_key_iv: "2b433d5864550767747da7c42063b1c2"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:54.150041 32130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:54.186158 32130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:54.188599 32130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:54.188753 32130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:54.189055 32130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:54.189445 32130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:54.189503 32130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.189544 32130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:54.189572 32130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.196314 32288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:44675 every 8 connection(s)
I20260924 19:28:54.196717 32130 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:44675
I20260924 19:28:54.197192 32130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:28:54.205086 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 32130
I20260924 19:28:54.205395 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:38537
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:43225
--webserver_port=36269
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:54.210911 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.197955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43225 (local address 127.18.229.129:42887)
0924 19:28:54.198453 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:54.198499 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:54.199145 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:54.199869 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:54.199901 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:54.200164 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:54.200450 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:54.200457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.202251 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.202252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:54.203533 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:54.203536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.204401 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.204403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:54.204538 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.205146 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:54.205156 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:54.206794 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:54.209123 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:54.209130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:54.209133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:54.209418 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:54.209716 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:54.209717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:54.209717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:54.209853 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:54.210289 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:54.210295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:54.210430 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:28:54.210539 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:28:54.210699 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":89,"threads_started":1}
I20260924 19:28:54.210901 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.198364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:42887 (local address 127.18.229.190:43225)
0924 19:28:54.199368 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:54.199378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:54.199419 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:28:54.199583 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:54.199590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:54.199650 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:54.200600 (+ 950us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:54.200710 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.202131 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.203620 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.204283 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.204406 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.206920 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:54.206924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:54.206924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:54.206937 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:54.209000 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.209520 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.209523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.209523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.209631 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.209932 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.209933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.209933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.210200 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:54.210310 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:54.210589 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:28:54.210699 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":79,"threads_started":1}
I20260924 19:28:54.211846 32289 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43225
I20260924 19:28:54.211985 32289 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.212219 32289 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
W20260924 19:28:54.213107 32289 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:28:54.213179 32289 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.213240 32289 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:54.213660 32289 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.213730 32289 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
W20260924 19:28:54.214082 32289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:28:54.309793 32293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:54.310007 32293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:54.310047 32293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:54.311265 32293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:54.311367 32293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:54.311437 32293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:28:54.313058 32293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:38537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=36269
--tserver_master_addrs=127.18.229.190:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:54.313308 32293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:54.313521 32293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:54.318431 32298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:54.318549 32299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:54.318420 32301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:54.319149 32293 server_base.cc:1034] running on GCE node
I20260924 19:28:54.319358 32293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:54.319625 32293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:54.320772 32293 hybrid_clock.cc:648] HybridClock initialized: now 1790278134320749 us; error 39 us; skew 500 ppm
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:54.323331 32293 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:28:54.323794 32293 webserver.cc:460] Webserver started at http://127.18.229.130:36269/ using document root <none> and password file <none>
I20260924 19:28:54.324028 32293 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:54.324101 32293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:54.340567 32293 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:28:54.346027 32309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.346211 32293 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 19:28:54.346319 32293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "04af23abdd7844ecab84ad8e49a9f9c4"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "6ce3c5973c062cb7151fad456266ae90"
server_key_iv: "92a6d1d2bd4fd57463335850304e461a"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:54.346736 32293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:54.372008 32293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:54.374056 32293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:54.374202 32293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:54.374469 32293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:54.374832 32293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:54.374867 32293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.374889 32293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:54.374902 32293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.381910 32293 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:38537
I20260924 19:28:54.381999 32422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:38537 every 8 connection(s)
I20260924 19:28:54.382455 32293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:54.392407 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 32293
I20260924 19:28:54.392707 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.229.148:37797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:45865
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:43225
--webserver_port=44007
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:28:54.396529 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.384899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:35495 (local address 127.18.229.190:43225)
0924 19:28:54.384958 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:54.384960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:54.384976 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:28:54.385583 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:54.385585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:54.385599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:54.385629 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:54.386337 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.386963 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.388491 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.388803 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.388852 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.392942 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:54.392944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:54.392945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:54.392953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:54.394840 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.395351 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.395352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.395353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.395427 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.395728 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.395729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.395729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.395866 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:54.395911 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:54.396332 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 19:28:54.396420 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:28:54.397023 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.384485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43225 (local address 127.18.229.130:35495)
0924 19:28:54.384851 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:54.384896 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:54.385453 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:54.385820 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:54.385823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:54.385992 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:54.386228 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:54.386234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.387100 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.387101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:54.388372 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:54.388375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.388961 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.388962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:54.389061 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.389589 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:54.389598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:54.392839 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:54.394951 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:54.394958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:54.394960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:54.395257 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:54.395514 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:54.395515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:54.395515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:54.395646 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:54.395959 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:54.395966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:54.396177 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 19:28:54.396621 (+ 444us) client_negotiation.cc:240] Negotiation successful
0924 19:28:54.396791 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":79,"threads_started":1}
I20260924 19:28:54.397565 32423 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43225
I20260924 19:28:54.397653 32423 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.397838 32423 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
W20260924 19:28:54.398469 32423 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:28:54.398535 32423 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.398591 32423 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:54.398882 32423 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.398950 32423 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
W20260924 19:28:54.399233 32423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:28:54.495921 32426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:54.496313 32426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:54.496366 32426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:54.497520 32426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:54.497581 32426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:54.497627 32426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:28:54.499233 32426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.229.148:37797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:45865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=44007
--tserver_master_addrs=127.18.229.190:43225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:54.499490 32426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:54.499715 32426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:54.504858 32432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:54.504874 32435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:54.505146 32433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:54.505218 32426 server_base.cc:1034] running on GCE node
I20260924 19:28:54.505425 32426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:54.505702 32426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:54.506876 32426 hybrid_clock.cc:648] HybridClock initialized: now 1790278134506863 us; error 37 us; skew 500 ppm
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:54.509836 32426 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:28:54.510249 32426 webserver.cc:460] Webserver started at http://127.18.229.131:44007/ using document root <none> and password file <none>
I20260924 19:28:54.510470 32426 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:54.510550 32426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:54.521448 32426 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:28:54.522367 32443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.522612 32426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:28:54.522715 32426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "ed71fb9161574bff924631db1fc9b4bc"
format_stamp: "Formatted at 2026-09-24 19:28:53 on dist-test-slave-r8cc"
server_key: "dfb276bb74b7b01d84279ab4d161c050"
server_key_iv: "dbf48cc18ed83ae78c70d4431e8359ac"
server_key_version: "kuduclusterkey@0"
I20260924 19:28:54.523238 32426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:54.538220 32426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:54.540577 32426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:54.540750 32426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:54.541013 32426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:28:54.541379 32426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:28:54.541447 32426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.541481 32426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:28:54.541509 32426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:54.548831 32426 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:45865
I20260924 19:28:54.549360 32426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:28:54.549624 32556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:45865 every 8 connection(s)
Sep 24 19:28:54 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278134, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:54.557163 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 32426
I20260924 19:28:54.563920 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.550429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:41057 (local address 127.18.229.190:43225)
0924 19:28:54.550475 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:54.550477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:54.550705 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:28:54.551359 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:54.551360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:54.551371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:54.551412 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:54.552211 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.552887 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.554178 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.554408 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.554467 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.560553 (+ 6086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:54.560554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:54.560554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:54.560563 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:54.562321 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.562805 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.562806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.562807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.562877 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.563178 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.563178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.563179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.563313 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:54.563383 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:54.563735 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 19:28:54.563821 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:28:54.564461 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.550408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43225 (local address 127.18.229.131:41057)
0924 19:28:54.550569 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:54.550615 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:54.551240 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:54.551509 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:54.551532 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:54.551764 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:54.552067 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:54.552074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.552996 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.552997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:54.554093 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:54.554096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.554490 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.554491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:54.554619 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.555236 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:54.555246 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:54.560432 (+ 5186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:54.562405 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:54.562412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:54.562415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:54.562720 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:54.562950 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:54.562951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:54.562951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:54.563104 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:54.563384 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:54.563391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:54.563558 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 19:28:54.564024 (+ 466us) client_negotiation.cc:240] Negotiation successful
0924 19:28:54.564231 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 19:28:54.564993 32557 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43225
I20260924 19:28:54.565092 32557 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.565297 32557 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
W20260924 19:28:54.566025 32557 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:28:54.566118 32557 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.566198 32557 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:54.566845 32557 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:54.566929 32557 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
W20260924 19:28:54.567195 32557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:28:54.568576 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:54.557594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.18.229.190:43225)
0924 19:28:54.557733 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:54.557735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:54.560398 (+ 2663us) server_negotiation.cc:407] Connection header received
0924 19:28:54.560480 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:54.560481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:54.560494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:54.560560 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:54.561585 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.562394 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.563699 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:54.563912 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:54.564047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:54.565713 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:54.565714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:54.565714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:54.565722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:54.567458 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.567765 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.567766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.567766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.567827 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:54.568034 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:54.568035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:54.568036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:54.568199 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:54.568360 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:54.568401 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 19:28:54.568485 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
Sep 24 19:28:55 dist-test-slave-r8cc krb5kdc[31095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278135, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:55.214262 32289 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:55.214421 32289 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:55.214862 32091 ts_manager.cc:194] Registered new tserver with Master: dbdb476103b84f4c9fb3dc1873bd5435 (127.18.229.129:44675)
I20260924 19:28:55.215853 32091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:42887
I20260924 19:28:55.399390 32423 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:55.399525 32423 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:55.399955 32091 ts_manager.cc:194] Registered new tserver with Master: 04af23abdd7844ecab84ad8e49a9f9c4 (127.18.229.130:38537)
I20260924 19:28:55.400516 32091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:35495
Sep 24 19:28:55 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278135, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
I20260924 19:28:55.567378 32557 heartbeater.cc:464] Registering TS with master...
I20260924 19:28:55.567548 32557 heartbeater.cc:511] Master 127.18.229.190:43225 requested a full tablet report, sending...
I20260924 19:28:55.567868 32091 ts_manager.cc:194] Registered new tserver with Master: ed71fb9161574bff924631db1fc9b4bc (127.18.229.131:45865)
I20260924 19:28:55.568426 32091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:41057
I20260924 19:28:55.579007 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:28:55.597031 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.583929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.18.229.190:43225)
0924 19:28:55.584407 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.584409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.584417 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:28:55.584452 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.584453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.584465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.585290 (+ 825us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:55.585330 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.586118 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.587201 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.587521 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.588305 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.593587 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:55.593589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:55.593589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:55.593596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:55.595422 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:55.595828 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:55.595829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:55.595829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:55.595891 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:55.596132 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:55.596133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:55.596133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:55.596268 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:55.596808 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.596850 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.596931 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":77,"threads_started":1}
I20260924 19:28:55.599390 32091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:28:55.601733 32091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:28:55.611797 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.605783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38537 (local address 127.0.0.1:39492)
0924 19:28:55.606658 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.606702 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:55.606766 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:55.607240 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:55.607244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:55.607246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:55.607312 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.607314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.608910 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.608911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.610146 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.610148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.611505 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.611506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.611588 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.611603 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:28:55.611657 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:28:55.611709 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":69,"threads_started":1}
I20260924 19:28:55.612618 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.605884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.18.229.130:38537)
0924 19:28:55.607068 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.607073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.607083 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:28:55.607117 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.607119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.607135 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.607379 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:55.607465 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.608813 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.610251 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.611388 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.612392 (+ 1004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.612469 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.612498 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.612537 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":57,"threads_started":1}
I20260924 19:28:55.613480 32357 tablet_service.cc:1467] Processing CreateTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e (DEFAULT_TABLE table=test-table [id=ee298672549f46a8856d8547111b5e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:28:55.613859 32357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876dc2d306fa452c84cdccbc03b4b02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:55.618180 32591 tablet_bootstrap.cc:492] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: Bootstrap starting.
I20260924 19:28:55.618918 32591 tablet_bootstrap.cc:654] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:55.619315 32591 log.cc:824] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:55.619937 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.606186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:44675 (local address 127.0.0.1:37364)
0924 19:28:55.611871 (+ 5685us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.611881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:55.611942 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:55.612384 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:55.612385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:55.612387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:55.612419 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.612420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.615159 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.615160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.616176 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.616179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.619688 (+ 3509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.619689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.619755 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.619759 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:28:55.619819 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:28:55.619862 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5675,"thread_start_us":60,"threads_started":1}
I20260924 19:28:55.622558 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.605991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45865 (local address 127.0.0.1:35926)
0924 19:28:55.606887 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.606896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:55.606933 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:55.613895 (+ 6962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:55.613896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:55.613897 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:55.613943 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.613944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.619835 (+ 5891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.619836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.620947 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.620951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.622316 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.622317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.622393 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.622397 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:28:55.622429 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 19:28:55.622486 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":60,"threads_started":1}
I20260924 19:28:55.623112 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.606101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.18.229.131:45865)
0924 19:28:55.613713 (+ 7612us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.613717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.613725 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:28:55.613757 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.613759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.613775 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.614004 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:55.614093 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.615042 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.621406 (+ 6364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.622202 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.622898 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.622966 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.622994 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.623036 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7575,"thread_start_us":41,"threads_started":1}
I20260924 19:28:55.623673 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.606282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.18.229.129:44675)
0924 19:28:55.612193 (+ 5911us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.612198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.612211 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:28:55.612247 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.612249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.612265 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.612481 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:55.612566 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.613579 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.616278 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.617035 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.619693 (+ 2658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.619764 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.623551 (+ 3787us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.623598 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5875,"thread_start_us":44,"threads_started":1}
I20260924 19:28:55.623947 32491 tablet_service.cc:1467] Processing CreateTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e (DEFAULT_TABLE table=test-table [id=ee298672549f46a8856d8547111b5e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:28:55.624307 32491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876dc2d306fa452c84cdccbc03b4b02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:55.624421 32216 tablet_service.cc:1467] Processing CreateTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e (DEFAULT_TABLE table=test-table [id=ee298672549f46a8856d8547111b5e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:28:55.624745 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876dc2d306fa452c84cdccbc03b4b02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:55.627986 32593 tablet_bootstrap.cc:492] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Bootstrap starting.
I20260924 19:28:55.628739 32593 tablet_bootstrap.cc:654] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:55.629124 32593 log.cc:824] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:55.629489 32594 tablet_bootstrap.cc:492] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: Bootstrap starting.
I20260924 19:28:55.630147 32594 tablet_bootstrap.cc:654] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:55.630483 32594 log.cc:824] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:55.631258 32593 tablet_bootstrap.cc:492] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: No bootstrap required, opened a new log
I20260924 19:28:55.631263 32591 tablet_bootstrap.cc:492] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: No bootstrap required, opened a new log
I20260924 19:28:55.631330 32591 ts_tablet_manager.cc:1397] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260924 19:28:55.631426 32593 ts_tablet_manager.cc:1397] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:28:55.633623 32591 raft_consensus.cc:348] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.633805 32591 raft_consensus.cc:374] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:55.633849 32591 raft_consensus.cc:729] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04af23abdd7844ecab84ad8e49a9f9c4, State: Initialized, Role: FOLLOWER
I20260924 19:28:55.633992 32591 consensus_queue.cc:260] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.634270 32591 ts_tablet_manager.cc:1428] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:28:55.634220 32593 raft_consensus.cc:348] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.634398 32423 heartbeater.cc:503] Master 127.18.229.190:43225 was elected leader, sending a full tablet report...
I20260924 19:28:55.634482 32593 raft_consensus.cc:374] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:55.634562 32593 raft_consensus.cc:729] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed71fb9161574bff924631db1fc9b4bc, State: Initialized, Role: FOLLOWER
I20260924 19:28:55.634738 32593 consensus_queue.cc:260] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.635010 32593 ts_tablet_manager.cc:1428] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 19:28:55.635147 32557 heartbeater.cc:503] Master 127.18.229.190:43225 was elected leader, sending a full tablet report...
I20260924 19:28:55.637475 32594 tablet_bootstrap.cc:492] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: No bootstrap required, opened a new log
I20260924 19:28:55.637614 32594 ts_tablet_manager.cc:1397] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 19:28:55.640012 32594 raft_consensus.cc:348] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.640240 32594 raft_consensus.cc:374] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:55.640311 32594 raft_consensus.cc:729] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdb476103b84f4c9fb3dc1873bd5435, State: Initialized, Role: FOLLOWER
I20260924 19:28:55.640475 32594 consensus_queue.cc:260] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.640760 32594 ts_tablet_manager.cc:1428] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:28:55.640892 32289 heartbeater.cc:503] Master 127.18.229.190:43225 was elected leader, sending a full tablet report...
Sep 24 19:28:55 dist-test-slave-r8cc krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278135, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for HTTP/127.18.229.148@KRBTEST.COM
W20260924 19:28:55.699329 32290 tablet.cc:2369] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:28:55.801375 32558 tablet.cc:2369] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:28:55.883895 32424 tablet.cc:2369] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:28:55.945451 32599 raft_consensus.cc:482] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:28:55.945601 32599 raft_consensus.cc:504] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.946040 32599 leader_election.cc:288] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04af23abdd7844ecab84ad8e49a9f9c4 (127.18.229.130:38537), ed71fb9161574bff924631db1fc9b4bc (127.18.229.131:45865)
I20260924 19:28:55.950184 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.946341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:54983 (local address 127.18.229.130:38537)
0924 19:28:55.946385 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.946387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.946558 (+ 171us) server_negotiation.cc:407] Connection header received
0924 19:28:55.946664 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.946666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.946676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.946704 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:55.946936 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.947681 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.949215 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.949788 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.949838 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.949847 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.950073 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.950114 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:28:55.950215 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.946654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:52431 (local address 127.18.229.131:45865)
0924 19:28:55.946699 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.946701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.946839 (+ 138us) server_negotiation.cc:407] Connection header received
0924 19:28:55.946947 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.946948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.946957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.946987 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:28:55.947203 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.947911 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.949277 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.949825 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.949874 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.949882 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.950105 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.950143 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:28:55.950451 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.946470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:45865 (local address 127.18.229.129:52431)
0924 19:28:55.946749 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.946759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:55.946867 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:55.947053 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:55.947053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:55.947055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:55.947122 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.947124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.948016 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.948017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.949170 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.949174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.949941 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.949942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.950026 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.950029 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:28:55.950324 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 19:28:55.950364 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":63,"threads_started":1}
I20260924 19:28:55.950831 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.946253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38537 (local address 127.18.229.129:54983)
0924 19:28:55.946474 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.946485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:55.946587 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:55.946775 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:55.946776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:55.946779 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:28:55.946854 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.946856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.947785 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.947786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.949112 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:55.949115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.949924 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.949925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:55.949992 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.949996 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:28:55.950710 (+ 714us) client_negotiation.cc:240] Negotiation successful
0924 19:28:55.950752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260924 19:28:55.950914 32511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876dc2d306fa452c84cdccbc03b4b02e" candidate_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed71fb9161574bff924631db1fc9b4bc" is_pre_election: true
I20260924 19:28:55.951061 32511 raft_consensus.cc:2454] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbdb476103b84f4c9fb3dc1873bd5435 in term 0.
I20260924 19:28:55.951300 32178 leader_election.cc:302] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbdb476103b84f4c9fb3dc1873bd5435, ed71fb9161574bff924631db1fc9b4bc; no voters:
I20260924 19:28:55.951516 32377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876dc2d306fa452c84cdccbc03b4b02e" candidate_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" is_pre_election: true
I20260924 19:28:55.951609 32599 raft_consensus.cc:2781] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:28:55.951673 32599 raft_consensus.cc:482] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:28:55.951695 32599 raft_consensus.cc:3037] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:55.951689 32377 raft_consensus.cc:2454] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbdb476103b84f4c9fb3dc1873bd5435 in term 0.
I20260924 19:28:55.952669 32599 raft_consensus.cc:504] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.952795 32599 leader_election.cc:288] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [CANDIDATE]: Term 1 election: Requested vote from peers 04af23abdd7844ecab84ad8e49a9f9c4 (127.18.229.130:38537), ed71fb9161574bff924631db1fc9b4bc (127.18.229.131:45865)
I20260924 19:28:55.952992 32377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876dc2d306fa452c84cdccbc03b4b02e" candidate_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04af23abdd7844ecab84ad8e49a9f9c4"
I20260924 19:28:55.953030 32511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876dc2d306fa452c84cdccbc03b4b02e" candidate_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed71fb9161574bff924631db1fc9b4bc"
I20260924 19:28:55.953068 32377 raft_consensus.cc:3037] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:55.953104 32511 raft_consensus.cc:3037] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:55.953956 32377 raft_consensus.cc:2454] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbdb476103b84f4c9fb3dc1873bd5435 in term 1.
I20260924 19:28:55.953953 32511 raft_consensus.cc:2454] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbdb476103b84f4c9fb3dc1873bd5435 in term 1.
I20260924 19:28:55.954126 32178 leader_election.cc:302] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04af23abdd7844ecab84ad8e49a9f9c4, dbdb476103b84f4c9fb3dc1873bd5435; no voters:
I20260924 19:28:55.954239 32599 raft_consensus.cc:2781] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:28:55.954391 32599 raft_consensus.cc:686] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 1 LEADER]: Becoming Leader. State: Replica: dbdb476103b84f4c9fb3dc1873bd5435, State: Running, Role: LEADER
I20260924 19:28:55.954504 32599 consensus_queue.cc:237] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } }
I20260924 19:28:55.955314 32091 catalog_manager.cc:5495] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbdb476103b84f4c9fb3dc1873bd5435 (127.18.229.129). New cstate: current_term: 1 leader_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbdb476103b84f4c9fb3dc1873bd5435" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 44675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 } health_report { overall_health: UNKNOWN } } }
I20260924 19:28:55.964143 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.961841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.18.229.129:44675)
0924 19:28:55.961890 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:55.961892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:55.961902 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:28:55.961961 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:55.961962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:55.961983 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:55.962057 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:28:55.962171 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.962628 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.963374 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:55.963564 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:55.963659 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:55.963808 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:28:55.963906 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:28:55.963994 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:55.964019 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:28:55.964037 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:28:55.965857 32377 raft_consensus.cc:1261] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 1 FOLLOWER]: Refusing update from remote peer dbdb476103b84f4c9fb3dc1873bd5435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:28:55.965867 32511 raft_consensus.cc:1261] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 1 FOLLOWER]: Refusing update from remote peer dbdb476103b84f4c9fb3dc1873bd5435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:28:55.966194 32611 consensus_queue.cc:1035] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed71fb9161574bff924631db1fc9b4bc" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:28:55.966327 32599 consensus_queue.cc:1035] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04af23abdd7844ecab84ad8e49a9f9c4" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:28:55.969252 32615 mvcc.cc:205] Tried to move back new op lower bound from 7332979244913467392 to 7332979244870189056. Current Snapshot: MvccSnapshot[applied={T|T < 7332979244913467392}]
I20260924 19:28:55.971431 32616 mvcc.cc:205] Tried to move back new op lower bound from 7332979244913467392 to 7332979244870189056. Current Snapshot: MvccSnapshot[applied={T|T < 7332979244913467392}]
I20260924 19:28:55.971540 32090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:28:55.971649 32090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:28:55.972417 32090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3be9aace2934d74b4486051710c3399: Sending DeleteTablet for 3 replicas of tablet 876dc2d306fa452c84cdccbc03b4b02e
I20260924 19:28:55.972700 32216 tablet_service.cc:1514] Processing DeleteTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:55 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:37364
I20260924 19:28:55.972743 32357 tablet_service.cc:1514] Processing DeleteTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:55 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39492
I20260924 19:28:55.972915 32356 tablet_service.cc:1514] Processing DeleteTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:55 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39492
I20260924 19:28:55.972920 32623 tablet_replica.cc:330] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435: stopping tablet replica
I20260924 19:28:55.973013 32623 raft_consensus.cc:2229] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:28:55.973095 32623 raft_consensus.cc:2258] T 876dc2d306fa452c84cdccbc03b4b02e P dbdb476103b84f4c9fb3dc1873bd5435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:28:55.973219 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 32130
I20260924 19:28:55.973198 32491 tablet_service.cc:1514] Processing DeleteTablet for tablet 876dc2d306fa452c84cdccbc03b4b02e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:28:55 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:35926
I20260924 19:28:55.978549 32624 tablet_replica.cc:330] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4: stopping tablet replica
I20260924 19:28:55.978693 32624 raft_consensus.cc:2229] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:28:55.978762 32624 raft_consensus.cc:2258] T 876dc2d306fa452c84cdccbc03b4b02e P 04af23abdd7844ecab84ad8e49a9f9c4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:28:55.978961 32075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:44675 (error 108)
I20260924 19:28:55.979642 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 32293
I20260924 19:28:55.980844 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.980711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:44675 (local address 127.0.0.1:37384)
0924 19:28:55.980769 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.980808 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 19:28:55.980978 32075 catalog_manager.cc:4584] TS dbdb476103b84f4c9fb3dc1873bd5435 (127.18.229.129:44675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 876dc2d306fa452c84cdccbc03b4b02e: Network error: Client connection negotiation failed: client connection to 127.18.229.129:44675: connect: Connection refused (error 111)
I20260924 19:28:55.982712 32625 tablet_replica.cc:330] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: stopping tablet replica
I20260924 19:28:55.982811 32625 raft_consensus.cc:2229] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:28:55.982939 32625 raft_consensus.cc:2258] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:28:55.983410 32625 ts_tablet_manager.cc:1905] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:28:55.985397 32625 ts_tablet_manager.cc:1918] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:28:55.985471 32625 log.cc:1196] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/876dc2d306fa452c84cdccbc03b4b02e
I20260924 19:28:55.985740 32625 ts_tablet_manager.cc:1939] T 876dc2d306fa452c84cdccbc03b4b02e P ed71fb9161574bff924631db1fc9b4bc: Deleting consensus metadata
I20260924 19:28:55.986150 32077 catalog_manager.cc:4841] TS ed71fb9161574bff924631db1fc9b4bc (127.18.229.131:45865): tablet 876dc2d306fa452c84cdccbc03b4b02e (table test-table [id=ee298672549f46a8856d8547111b5e89]) successfully deleted
I20260924 19:28:55.986907 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:55.986795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38537 (local address 127.0.0.1:39496)
0924 19:28:55.986844 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:55.986873 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:28:55.986943 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 32426
I20260924 19:28:55.991667 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 32058
I20260924 19:28:55.998543 19350 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:28:56.343886 19350 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:28:56.343981 19350 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:28:56.689368 19350 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:28:56.689 UTC [31125] LOG: received smart shutdown request
2026-09-24 19:28:56.693 UTC [31125] LOG: background worker "logical replication launcher" (PID 31134) exited with exit code 1
2026-09-24 19:28:56.694 UTC [31129] LOG: shutting down
2026-09-24 19:28:56.845 UTC [31125] LOG: database system is shut down
2026-09-24T19:28:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46232 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:28:57 dist-test-slave-r8cc krb5kdc[32635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:28:57 dist-test-slave-r8cc krb5kdc[32635](info): set up 2 sockets
Sep 24 19:28:57 dist-test-slave-r8cc krb5kdc[32635](info): commencing operation
krb5kdc: starting...
W20260924 19:28:57.983919 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:28:58.094651 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42575
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42575,127.18.229.189:41479,127.18.229.188:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:28:58.173765 32648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:58.174000 32648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:58.174036 32648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:58.175253 32648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:28:58.175323 32648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:58.175346 32648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:58.175366 32648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:28:58.175398 32648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:28:58.176914 32648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42575,127.18.229.189:41479,127.18.229.188:42087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:58.177183 32648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:58.177404 32648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:58.182193 32653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:58.182183 32654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:58.182183 32656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:58.182286 32648 server_base.cc:1034] running on GCE node
I20260924 19:28:58.182637 32648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:58.182922 32648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:58.184088 32648 hybrid_clock.cc:648] HybridClock initialized: now 1790278138184065 us; error 35 us; skew 500 ppm
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:58.186339 32648 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:28:58.186758 32648 webserver.cc:460] Webserver started at http://127.18.229.190:44943/ using document root <none> and password file <none>
I20260924 19:28:58.186988 32648 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:58.187050 32648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:58.187191 32648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:28:58.188297 32648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "2ac243b8a4bc4a169c4baa663a7299c1"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "40381e6fdff74f95d82fe1e4655409ba"
server_key_iv: "ada95898c9e4c0cecc9136397723c734"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.188637 32648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "2ac243b8a4bc4a169c4baa663a7299c1"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "40381e6fdff74f95d82fe1e4655409ba"
server_key_iv: "ada95898c9e4c0cecc9136397723c734"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.189941 32648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:28:58.190613 32663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:58.190778 32648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:28:58.190866 32648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "2ac243b8a4bc4a169c4baa663a7299c1"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "40381e6fdff74f95d82fe1e4655409ba"
server_key_iv: "ada95898c9e4c0cecc9136397723c734"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.190959 32648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:58.202858 32648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:58.204596 32648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:58.204746 32648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:58.208766 32648 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42575
I20260924 19:28:58.208812 32715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42575 every 8 connection(s)
I20260924 19:28:58.209117 32648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:28:58.209870 32716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:58.211360 32716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:58.219146 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 32648
I20260924 19:28:58.219259 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:28:58.219460 19350 external_mini_cluster.cc:1428] Setting key 6a123445f5dd65bff205cbce4f7e2390
I20260924 19:28:58.220091 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.212042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.18.229.190:42575)
0924 19:28:58.212232 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.212236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.212288 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:28:58.212771 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.212773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.212789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.212863 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.213341 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.214117 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.214927 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.215235 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.215307 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.217149 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.217155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.217156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.217161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.218635 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.218936 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.218938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.218938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.219022 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.219301 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.219302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.219302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.219535 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.219614 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.219828 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.219939 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":161,"thread_start_us":121,"threads_started":1}
I20260924 19:28:58.220306 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.212007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42575 (local address 127.0.0.1:56652)
0924 19:28:58.212181 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.212222 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.212653 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.212889 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.212891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.213118 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.213262 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.213266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.214195 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.214196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.214859 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.214860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.215301 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.215302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.215354 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.215750 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.215757 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.217077 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.218701 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.218705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.218707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.218873 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.219088 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.219089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.219089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.219197 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.219622 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.219628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.219706 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.219800 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.220009 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":51,"thread_start_us":69,"threads_started":1}
I20260924 19:28:58.220990 32716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:28:58.221338 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.221215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56246)
0924 19:28:58.221260 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.221309 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:28:58.221482 32666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
I20260924 19:28:58.221877 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.221782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56250)
0924 19:28:58.221825 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.221857 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:28:58.222016 32716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:41479: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.18.229.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab.
Entry for principal kudu/127.18.229.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab.
I20260924 19:28:58.242198 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:41479
--webserver_interface=127.18.229.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab
--principal=kudu/127.18.229.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42575,127.18.229.189:41479,127.18.229.188:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:28:58.271335 32716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } attempt: 1
I20260924 19:28:58.271834 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.271680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56262)
0924 19:28:58.271749 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.271797 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:28:58.272222 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.272137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56266)
0924 19:28:58.272171 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.272200 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 19:28:58.272327 32716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:41479: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
W20260924 19:28:58.325013 32723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:58.325217 32723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:58.325238 32723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:58.326583 32723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:28:58.326627 32723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:58.326642 32723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:58.326656 32723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:28:58.326669 32723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:28:58.328307 32723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42575,127.18.229.189:41479,127.18.229.188:42087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab
--principal=kudu/127.18.229.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.229.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:58.328553 32723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:58.328781 32723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:58.334028 32728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:58.334137 32723 server_base.cc:1034] running on GCE node
W20260924 19:28:58.334018 32731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:58.334019 32729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:58.334465 32723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:58.334714 32723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:58.335848 32723 hybrid_clock.cc:648] HybridClock initialized: now 1790278138335838 us; error 30 us; skew 500 ppm
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:58.338250 32723 init.cc:377] Logged in from keytab as kudu/127.18.229.189@KRBTEST.COM (short username kudu)
I20260924 19:28:58.338625 32723 webserver.cc:460] Webserver started at http://127.18.229.189:37295/ using document root <none> and password file <none>
I20260924 19:28:58.338796 32723 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:58.338835 32723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:58.338922 32723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:28:58.339898 32723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data/instance:
uuid: "e060bf6e9a4c44d893b0530f4c9df211"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "faf8355c3c4c99d24344c3f49ba820d6"
server_key_iv: "e2dbee302302aa786254f15246ac1ec9"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.340256 32723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal/instance:
uuid: "e060bf6e9a4c44d893b0530f4c9df211"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "faf8355c3c4c99d24344c3f49ba820d6"
server_key_iv: "e2dbee302302aa786254f15246ac1ec9"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.340478 32716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } attempt: 2
I20260924 19:28:58.340828 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.340717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56272)
0924 19:28:58.340769 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.340802 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:28:58.341140 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.341042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56286)
0924 19:28:58.341096 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.341119 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 19:28:58.341223 32716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:41479: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41479: connect: Connection refused (error 111)
I20260924 19:28:58.341682 32723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:28:58.342350 32738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:58.342507 32723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:28:58.342599 32723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal
uuid: "e060bf6e9a4c44d893b0530f4c9df211"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "faf8355c3c4c99d24344c3f49ba820d6"
server_key_iv: "e2dbee302302aa786254f15246ac1ec9"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.342664 32723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:58.365748 32723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:58.367576 32723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:58.367728 32723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:58.371482 32723 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.189:41479
I20260924 19:28:58.371536 322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.189:41479 every 8 connection(s)
I20260924 19:28:58.371884 32723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
I20260924 19:28:58.372707 323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:58.374157 323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:28:58.376755 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 32723
I20260924 19:28:58.376865 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-1/wal/instance
I20260924 19:28:58.377051 19350 external_mini_cluster.cc:1428] Setting key d0d21f761666b3f8696ee9deb1820afc
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:58.383589 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.374807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.18.229.190:42575)
0924 19:28:58.374863 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.374865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.375056 (+ 191us) server_negotiation.cc:407] Connection header received
0924 19:28:58.375503 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.375504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.375512 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.375559 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.376163 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.376634 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.378180 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.378378 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.378457 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.380639 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.380641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.380641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.380649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.382282 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.382656 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.382657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.382657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.382723 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.382945 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.382946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.382947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.383085 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.383157 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.383386 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.383475 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:28:58.383613 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.374776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42575 (local address 127.0.0.1:56658)
0924 19:28:58.374927 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.374967 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.375409 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.375619 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.375622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.375859 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.376044 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.376048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.376718 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.376720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.378047 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.378050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.378464 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.378465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.378599 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.379136 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.379145 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.380553 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.382363 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.382369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.382371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.382585 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.382796 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.382797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.382797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.382890 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.383152 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.383156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.383263 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.383344 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.383457 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260924 19:28:58.384354 323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.229.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.188@KRBTEST.COM" created.
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:28:58.391521 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.384613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56290)
0924 19:28:58.384668 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.384677 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.384710 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.384966 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.384967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.385101 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.385140 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.385141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.386007 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.386007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.386486 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.386487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.386950 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.386951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.387039 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.387367 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.387372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.388730 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.390370 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.390371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.390371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.390606 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.390796 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.390796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.390797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.390865 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.391117 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.391118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.391168 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.391248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.391359 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 19:28:58.391520 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.384685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.18.229.189:41479)
0924 19:28:58.384847 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.384851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.384865 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:28:58.384881 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.384882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.384892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.384967 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.385218 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.385918 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.386579 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.386868 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.386951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.388810 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.388817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.388817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.388825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.390291 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.390687 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.390689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.390690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.390738 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.390926 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.390927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.390927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.391041 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.391109 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.391281 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.391405 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260924 19:28:58.392285 323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:28:58.392637 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.392502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39226)
0924 19:28:58.392556 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.392607 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 19:28:58.392771 32739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
I20260924 19:28:58.393172 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.393066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39242)
0924 19:28:58.393111 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.393141 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:28:58.393301 323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:42087: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab.
Entry for principal kudu/127.18.229.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab.
I20260924 19:28:58.402416 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:42087
--webserver_interface=127.18.229.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab
--principal=kudu/127.18.229.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42575,127.18.229.189:41479,127.18.229.188:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:28:58.432358 32716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } attempt: 3
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:28:58.439682 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.432784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56306)
0924 19:28:58.432866 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.432879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.432941 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.433089 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.433090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.433235 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.433301 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.433303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.434239 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.434240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.434720 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.434721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.435170 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.435170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.435222 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.435695 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.435699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.437030 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.438769 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.438770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.438770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.438923 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.439133 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.439133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.439134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.439181 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.439456 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.439457 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.439492 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.439540 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.439613 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:28:58.439769 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.432848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.18.229.189:41479)
0924 19:28:58.432917 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.432919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.432931 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:28:58.432998 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.432999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.433013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.433072 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.433370 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.434163 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.434791 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.435101 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.435175 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.437119 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.437121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.437121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.437128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.438682 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.438993 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.438994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.438994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.439065 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.439236 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.439237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.439237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.439389 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.439473 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.439576 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.439669 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:28:58.440285 32716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:28:58.440625 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.440528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39244)
0924 19:28:58.440571 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.440598 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:28:58.440956 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.440882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39252)
0924 19:28:58.440919 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.440942 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:28:58.441056 32716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:42087: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
I20260924 19:28:58.442492 323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } attempt: 1
I20260924 19:28:58.442813 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.442704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39268)
0924 19:28:58.442750 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.442784 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:28:58.443126 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.443042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39276)
0924 19:28:58.443082 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.443106 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:28:58.443229 323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:42087: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
I20260924 19:28:58.472203 32716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } attempt: 1
I20260924 19:28:58.472680 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.472536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39284)
0924 19:28:58.472611 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.472649 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 19:28:58.473058 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.472933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39298)
0924 19:28:58.472987 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.473027 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:28:58.473207 32716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:42087: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
W20260924 19:28:58.484665 330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:28:58.484848 330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:28:58.484869 330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:28:58.485908 330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:28:58.485939 330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:28:58.485952 330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:28:58.485965 330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:28:58.485976 330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:28:58.487409 330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42575,127.18.229.189:41479,127.18.229.188:42087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab
--principal=kudu/127.18.229.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.229.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:28:58.487591 330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:28:58.487741 330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:28:58.492646 338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:28:58.492635 335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:58.492780 330 server_base.cc:1034] running on GCE node
W20260924 19:28:58.492636 336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:28:58.493067 330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:28:58.493409 330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:28:58.494565 330 hybrid_clock.cc:648] HybridClock initialized: now 1790278138494560 us; error 31 us; skew 500 ppm
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:28:58.496923 330 init.cc:377] Logged in from keytab as kudu/127.18.229.188@KRBTEST.COM (short username kudu)
I20260924 19:28:58.497313 330 webserver.cc:460] Webserver started at http://127.18.229.188:33985/ using document root <none> and password file <none>
I20260924 19:28:58.497526 330 fs_manager.cc:359] Metadata directory not provided
I20260924 19:28:58.497586 330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:28:58.497730 330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:28:58.498720 330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data/instance:
uuid: "659b3ed1765b4be68f6c1851c9954763"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "c1718ed34c940212f547c120e383ab71"
server_key_iv: "b38261a9bfafdd944291035243c58f08"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.499063 330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal/instance:
uuid: "659b3ed1765b4be68f6c1851c9954763"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "c1718ed34c940212f547c120e383ab71"
server_key_iv: "b38261a9bfafdd944291035243c58f08"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.500326 330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:28:58.500972 345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:28:58.501137 330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:28:58.501230 330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal
uuid: "659b3ed1765b4be68f6c1851c9954763"
format_stamp: "Formatted at 2026-09-24 19:28:58 on dist-test-slave-r8cc"
server_key: "c1718ed34c940212f547c120e383ab71"
server_key_iv: "b38261a9bfafdd944291035243c58f08"
server_key_version: "encryptionkey@0"
I20260924 19:28:58.501297 330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:28:58.511365 323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } attempt: 2
I20260924 19:28:58.511816 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.511684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39304)
0924 19:28:58.511738 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.511778 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:28:58.512439 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.512343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39312)
0924 19:28:58.512382 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.512412 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:28:58.512570 323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:42087: Network error: Client connection negotiation failed: client connection to 127.18.229.188:42087: connect: Connection refused (error 111)
I20260924 19:28:58.513358 330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:28:58.515018 330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:28:58.515169 330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:28:58.518989 330 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.188:42087
I20260924 19:28:58.519017 397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.188:42087 every 8 connection(s)
I20260924 19:28:58.519339 330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
I20260924 19:28:58.520160 398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:28:58.521584 398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:28:58.526628 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 330
I20260924 19:28:58.526726 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790277987605586-19350-0/minicluster-data/master-2/wal/instance
I20260924 19:28:58.526932 19350 external_mini_cluster.cc:1428] Setting key eb5ba4f966be2838df6deb0ac9a9815b
I20260924 19:28:58.529821 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.522165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.18.229.190:42575)
0924 19:28:58.522241 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.522243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.522390 (+ 147us) server_negotiation.cc:407] Connection header received
0924 19:28:58.522807 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.522808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.522817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.522868 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.523391 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.523845 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.524769 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.524941 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.524993 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.526935 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.526937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.526937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.526945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.528548 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.528959 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.528959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.528960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.529007 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.529245 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.529246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.529246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.529337 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.529385 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.529650 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.529728 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:28:58.529851 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.522127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42575 (local address 127.0.0.1:56666)
0924 19:28:58.522286 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.522318 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.522733 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.522917 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.522920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.523148 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.523322 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.523326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.523910 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.523911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.524706 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.524708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.525005 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.525006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.525106 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.525566 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.525574 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.526838 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.528613 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.528619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.528620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.528890 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.529072 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.529072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.529073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.529189 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.529396 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.529400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.529531 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.529611 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.529714 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 19:28:58.530750 398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:28:58.535770 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.527257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.18.229.188:42087)
0924 19:28:58.527472 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.527475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.527514 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:28:58.527599 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.527600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.527610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.527669 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.528401 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.529373 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.530279 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.530620 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.530669 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.533026 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.533035 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.533036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.533043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.534590 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.534920 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.534923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.534924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.534977 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.535177 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.535178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.535178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.535344 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.535494 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.535563 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.535652 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
I20260924 19:28:58.537348 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.531067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.18.229.189:41479)
0924 19:28:58.531113 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.531115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.531124 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:28:58.531148 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.531149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.531158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.531209 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.531502 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.531937 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.532683 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.532983 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.533047 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.534736 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.534737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.534738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.534743 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.536238 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.536577 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.536578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.536578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.536633 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.536883 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.536884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.536885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.537004 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.537060 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.537217 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.537293 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:28:58.537395 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.530980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41479 (local address 127.0.0.1:56322)
0924 19:28:58.531029 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.531040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.531080 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.531226 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.531227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.531371 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.531428 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.531429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.532013 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.532014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.532545 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.532546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.533045 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.533046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.533117 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.533419 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.533422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.534658 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.536303 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.536304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.536304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.536488 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.536691 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.536692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.536692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.536837 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.537073 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.537074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.537116 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.537194 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.537298 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:28:58.537710 398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:28:58.543834 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.536800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.18.229.190:42575)
0924 19:28:58.536839 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.536841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.536848 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:28:58.536867 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.536868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.536877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.536928 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.537509 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.538167 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.538888 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.539087 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.539132 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.541331 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.541332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.541332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.541339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.542746 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.543103 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.543104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.543104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.543148 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.543306 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.543307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.543307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.543425 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.543606 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.543636 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.543724 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:28:58.544286 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.537899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39320)
0924 19:28:58.537931 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.537939 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.537968 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.538091 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.538092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.538231 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.538264 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.538265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.538831 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.538832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.539470 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.539473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.539801 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.539802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.539869 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.540221 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.540226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.541614 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.543307 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.543308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.543309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.543498 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.543690 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.543691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.543691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.543752 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.544026 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.544028 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.544079 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.544175 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.544236 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 19:28:58.544412 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.537956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.18.229.188:42087)
0924 19:28:58.537987 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.537989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.537998 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:28:58.538015 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.538016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.538024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.538085 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.538333 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.538760 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.539545 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.539734 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.539783 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.541710 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.541711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.541711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.541717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.543230 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.543569 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.543570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.543571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.543624 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.543809 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.543810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.543810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.543954 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.544021 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.544234 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.544318 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:28:58.545965 398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: Bootstrap starting.
I20260924 19:28:58.546953 398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:58.547279 398 log.cc:824] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:58.548059 398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: No bootstrap required, opened a new log
I20260924 19:28:58.548327 32716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } attempt: 2
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:28:58.549757 398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.549866 398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:58.549887 398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659b3ed1765b4be68f6c1851c9954763, State: Initialized, Role: FOLLOWER
I20260924 19:28:58.550050 398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.550289 404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.550346 398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:28:58.550362 404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:28:58.551697 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.544423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.18.229.189:41479)
0924 19:28:58.544643 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.544645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.544651 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:28:58.544668 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.544668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.544675 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.544733 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.545286 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.545752 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.546629 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.546795 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.546845 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.549190 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.549191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.549191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.549198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.550611 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.550926 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.550927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.550927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.550981 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.551163 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.551164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.551165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.551282 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.551477 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.551540 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.551617 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:28:58.552786 330 master_runner.cc:186] Error getting master registration for 127.18.229.190:42575: OK, instance_id { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" instance_seqno: 1790278138207475 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:28:58.555655 330 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:28:58.552021 415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:28:58.555835 415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:28:58.556416 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:28:58.557988 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.548541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39322)
0924 19:28:58.548582 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.548591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.548629 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.551741 (+ 3112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.551742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.551892 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.551945 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.551946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.552853 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.552854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.553376 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.553378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.553688 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.553690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.553743 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.554129 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.554134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.555164 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.556917 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.556918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.556919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.557140 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.557346 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.557347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.557347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.557411 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.557723 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.557724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.557764 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.557810 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.557894 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:28:58.558568 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.548606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.18.229.188:42087)
0924 19:28:58.551610 (+ 3004us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.551612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.551619 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:28:58.551638 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.551639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.551647 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.552023 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.552054 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.552742 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.553448 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.553619 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.553689 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.555253 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.555254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.555255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.555261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.556834 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.557218 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.557219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.557219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.557275 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.557489 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.557491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.557491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.557640 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.558367 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.558408 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.558482 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2986}
I20260924 19:28:58.560563 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: Bootstrap starting.
I20260924 19:28:58.561255 32716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:58.561540 32716 log.cc:824] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:58.565938 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.558280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.18.229.189:41479)
0924 19:28:58.558448 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.558450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.558670 (+ 220us) server_negotiation.cc:407] Connection header received
0924 19:28:58.558758 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.558760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.558771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.558802 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.559454 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.560191 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.561196 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.561475 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.561515 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.562921 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.562923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.562923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.562930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.564546 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.565132 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.565133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.565134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.565191 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.565400 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.565401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.565401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.565510 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.565574 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.565777 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.565851 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260924 19:28:58.568647 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.558222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.18.229.190:42575)
0924 19:28:58.558319 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.558321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.558329 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:28:58.558349 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.558350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.558358 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.558407 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.561540 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.562567 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.563785 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.564078 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.564138 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.565710 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.565711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.565711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.565719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.567339 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.567759 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.567761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.567761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.567813 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.568017 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.568018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.568018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.568183 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.568261 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.568491 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.568572 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:28:58.568758 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: No bootstrap required, opened a new log
I20260924 19:28:58.569211 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.558043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.18.229.188:42087)
0924 19:28:58.558336 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.558338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.558963 (+ 625us) server_negotiation.cc:407] Connection header received
0924 19:28:58.558983 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.558984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.558993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.559058 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.559633 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.560396 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.563424 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.563638 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.563714 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.565683 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.565684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.565684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.565691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.567988 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.568408 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.568409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.568410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.568462 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.568643 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.568644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.568644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.568754 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.568830 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.569026 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.569112 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":73,"threads_started":1}
I20260924 19:28:58.570613 32716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.570719 32716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:58.570740 32716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac243b8a4bc4a169c4baa663a7299c1, State: Initialized, Role: FOLLOWER
I20260924 19:28:58.570873 32716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.571192 32716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:28:58.571149 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.571250 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:28:58.572960 437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:28:58.573078 437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:28:58.573732 32648 master_runner.cc:186] Error getting master registration for 127.18.229.189:41479: OK, instance_id { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" instance_seqno: 1790278138370472 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:28:58.578742 32648 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:28:58.603721 323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } attempt: 3
Sep 24 19:28:58 dist-test-slave-r8cc krb5kdc[32635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278138, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:28:58.612656 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.604008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:42087 (local address 127.0.0.1:39342)
0924 19:28:58.604094 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 19:28:58.604139 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:28:58.604179 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:28:58.604427 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:28:58.604428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:28:58.604637 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:28:58.604686 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.604689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.605647 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.605649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.606465 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:28:58.606467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.606975 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.606976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:28:58.607039 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.607543 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:28:58.607549 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:28:58.609167 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:28:58.611547 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.611547 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.611548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.611764 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.611979 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:28:58.611980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:28:58.611981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:28:58.612047 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:28:58.612338 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:28:58.612338 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:28:58.612386 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:28:58.612459 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:28:58.612554 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:28:58.612674 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:28:58.604253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.18.229.188:42087)
0924 19:28:58.604294 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:28:58.604296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:28:58.604303 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:28:58.604324 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:28:58.604325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:28:58.604336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:28:58.604383 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:28:58.604771 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.605559 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.606556 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:28:58.606896 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:28:58.606967 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:28:58.609247 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:28:58.609249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:28:58.609249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:28:58.609256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:28:58.610924 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.611837 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.611838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.611838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.611900 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:28:58.612138 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:28:58.612139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:28:58.612139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:28:58.612271 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:28:58.612340 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:28:58.612491 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 19:28:58.612582 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:28:58.614589 323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: Bootstrap starting.
I20260924 19:28:58.615319 323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: Neither blocks nor log segments found. Creating new log.
I20260924 19:28:58.615628 323 log.cc:824] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: Log is configured to *not* fsync() on all Append() calls
I20260924 19:28:58.616518 323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: No bootstrap required, opened a new log
I20260924 19:28:58.618455 323 raft_consensus.cc:348] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.618599 323 raft_consensus.cc:374] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:28:58.618633 323 raft_consensus.cc:729] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e060bf6e9a4c44d893b0530f4c9df211, State: Initialized, Role: FOLLOWER
I20260924 19:28:58.618799 323 consensus_queue.cc:260] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.619127 323 sys_catalog.cc:564] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:28:58.620915 451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:28:58.621016 451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:28:58.619091 440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.622291 440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:28:58.623245 32723 master_runner.cc:263] No leader master found from master e060bf6e9a4c44d893b0530f4c9df211
I20260924 19:28:58.623291 32723 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:28:58.705518 32669 tablet.cc:2369] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:28:58.765715 351 tablet.cc:2369] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:28:58.868577 32744 tablet.cc:2369] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:28:58.953025 404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:28:58.953150 404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.953358 404 leader_election.cc:288] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac243b8a4bc4a169c4baa663a7299c1 (127.18.229.190:42575), e060bf6e9a4c44d893b0530f4c9df211 (127.18.229.189:41479)
I20260924 19:28:58.953644 32690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "659b3ed1765b4be68f6c1851c9954763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" is_pre_election: true
I20260924 19:28:58.953672 32765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "659b3ed1765b4be68f6c1851c9954763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e060bf6e9a4c44d893b0530f4c9df211" is_pre_election: true
I20260924 19:28:58.953805 32690 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659b3ed1765b4be68f6c1851c9954763 in term 0.
I20260924 19:28:58.953805 32765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659b3ed1765b4be68f6c1851c9954763 in term 0.
I20260924 19:28:58.953992 348 leader_election.cc:302] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659b3ed1765b4be68f6c1851c9954763, e060bf6e9a4c44d893b0530f4c9df211; no voters:
I20260924 19:28:58.954130 404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:28:58.954190 404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:28:58.954213 404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:58.954906 404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.955026 404 leader_election.cc:288] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [CANDIDATE]: Term 1 election: Requested vote from peers 2ac243b8a4bc4a169c4baa663a7299c1 (127.18.229.190:42575), e060bf6e9a4c44d893b0530f4c9df211 (127.18.229.189:41479)
I20260924 19:28:58.955166 32690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "659b3ed1765b4be68f6c1851c9954763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac243b8a4bc4a169c4baa663a7299c1"
I20260924 19:28:58.955196 32765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "659b3ed1765b4be68f6c1851c9954763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e060bf6e9a4c44d893b0530f4c9df211"
I20260924 19:28:58.955262 32690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:58.955271 32765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:28:58.955976 32765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659b3ed1765b4be68f6c1851c9954763 in term 1.
I20260924 19:28:58.956151 32690 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659b3ed1765b4be68f6c1851c9954763 in term 1.
I20260924 19:28:58.956162 348 leader_election.cc:302] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659b3ed1765b4be68f6c1851c9954763, e060bf6e9a4c44d893b0530f4c9df211; no voters:
I20260924 19:28:58.956295 404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:28:58.956422 404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [term 1 LEADER]: Becoming Leader. State: Replica: 659b3ed1765b4be68f6c1851c9954763, State: Running, Role: LEADER
I20260924 19:28:58.956511 404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } }
I20260924 19:28:58.956980 453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 659b3ed1765b4be68f6c1851c9954763. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.957070 453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:58.957268 455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:28:58.957451 455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:28:58.958806 32690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [term 1 FOLLOWER]: Refusing update from remote peer 659b3ed1765b4be68f6c1851c9954763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:28:58.958806 32765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [term 1 FOLLOWER]: Refusing update from remote peer 659b3ed1765b4be68f6c1851c9954763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:28:58.959105 453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:28:58.959208 404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:28:58.960302 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 659b3ed1765b4be68f6c1851c9954763. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.960302 440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 659b3ed1765b4be68f6c1851c9954763. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.960391 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:28:58.960391 440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:28:58.961246 404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.961328 404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:58.961416 440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.961493 440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:28:58.961565 455 catalog_manager.cc:1347] Generated new cluster ID: 1357d0e005164eadb7d57eb4d77bf195
I20260924 19:28:58.961608 455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:28:58.961673 457 mvcc.cc:205] Tried to move back new op lower bound from 7332979257172942848 to 7332979257166540800. Current Snapshot: MvccSnapshot[applied={T|T < 7332979257172942848 or (T in {7332979257172942848})}]
I20260924 19:28:58.961838 463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.961947 463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763 [sys.catalog]: This master's current role is: LEADER
I20260924 19:28:58.962105 426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "659b3ed1765b4be68f6c1851c9954763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac243b8a4bc4a169c4baa663a7299c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42575 } } peers { permanent_uuid: "e060bf6e9a4c44d893b0530f4c9df211" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41479 } } peers { permanent_uuid: "659b3ed1765b4be68f6c1851c9954763" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 42087 } } }
I20260924 19:28:58.962162 458 mvcc.cc:205] Tried to move back new op lower bound from 7332979257172942848 to 7332979257166540800. Current Snapshot: MvccSnapshot[applied={T|T < 7332979257172942848}]
I20260924 19:28:58.962177 426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:28:58.961918 456 mvcc.cc:205] Tried to move back new op lower bound from 7332979257172942848 to 7332979257166540800. Current Snapshot: MvccSnapshot[applied={T|T < 7332979257172942848 or (T in {7332979257172942848})}]
I20260924 19:28:58.986331 455 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:28:58.986986 455 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:28:58.999346 455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 659b3ed1765b4be68f6c1851c9954763: Generated new TSK 0
I20260924 19:28:58.999641 455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:28:59.573545 437 catalog_manager.cc:1259] Loaded cluster ID: 1357d0e005164eadb7d57eb4d77bf195
I20260924 19:28:59.573647 437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: loading cluster ID for follower catalog manager: success
I20260924 19:28:59.574452 437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: acquiring CA information for follower catalog manager: success
I20260924 19:28:59.574764 437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ac243b8a4bc4a169c4baa663a7299c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:28:59.621465 451 catalog_manager.cc:1259] Loaded cluster ID: 1357d0e005164eadb7d57eb4d77bf195
I20260924 19:28:59.621578 451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: loading cluster ID for follower catalog manager: success
I20260924 19:28:59.622388 451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: acquiring CA information for follower catalog manager: success
I20260924 19:28:59.622686 451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e060bf6e9a4c44d893b0530f4c9df211: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:29:00.048694 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 32648
I20260924 19:29:00.054148 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 32723
I20260924 19:29:00.058708 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 330
2026-09-24T19:29:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3106 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:29:00 dist-test-slave-r8cc krb5kdc[470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:29:00 dist-test-slave-r8cc krb5kdc[470](info): set up 2 sockets
Sep 24 19:29:00 dist-test-slave-r8cc krb5kdc[470](info): commencing operation
krb5kdc: starting...
W20260924 19:29:01.084322 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:29:01 dist-test-slave-r8cc krb5kdc[470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:29:01.200819 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:36693
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:36693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:01.280956 483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:01.281211 483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:01.281247 483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:01.282398 483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:29:01.282467 483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:01.282491 483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:01.282523 483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:29:01.282548 483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:29:01.284008 483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:36693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:36693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:01.284313 483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:01.284540 483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:01.289268 491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:01.289278 489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:01.289278 488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:01.289618 483 server_base.cc:1034] running on GCE node
I20260924 19:29:01.289839 483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:01.290158 483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:01.291317 483 hybrid_clock.cc:648] HybridClock initialized: now 1790278141291306 us; error 27 us; skew 500 ppm
Sep 24 19:29:01 dist-test-slave-r8cc krb5kdc[470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278141, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:01.296674 483 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:29:01.297132 483 webserver.cc:460] Webserver started at http://127.18.229.190:46739/ using document root <none> and password file <none>
I20260924 19:29:01.297366 483 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:01.297410 483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:01.297569 483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:01.298621 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "6877d1d28d8d4efc944a845fbd77ab6f"
format_stamp: "Formatted at 2026-09-24 19:29:01 on dist-test-slave-r8cc"
server_key: "0cb864baa8d215e8805bd600c801e12d"
server_key_iv: "15ac6441aec029da4627c50ef2325fb0"
server_key_version: "encryptionkey@0"
I20260924 19:29:01.298977 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "6877d1d28d8d4efc944a845fbd77ab6f"
format_stamp: "Formatted at 2026-09-24 19:29:01 on dist-test-slave-r8cc"
server_key: "0cb864baa8d215e8805bd600c801e12d"
server_key_iv: "15ac6441aec029da4627c50ef2325fb0"
server_key_version: "encryptionkey@0"
I20260924 19:29:01.300282 483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 19:29:01.300940 498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:01.301113 483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:01.301195 483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "6877d1d28d8d4efc944a845fbd77ab6f"
format_stamp: "Formatted at 2026-09-24 19:29:01 on dist-test-slave-r8cc"
server_key: "0cb864baa8d215e8805bd600c801e12d"
server_key_iv: "15ac6441aec029da4627c50ef2325fb0"
server_key_version: "encryptionkey@0"
I20260924 19:29:01.301249 483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:01.309532 483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:01.311146 483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:01.311247 483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:01.315111 483 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:36693
I20260924 19:29:01.315141 550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:36693 every 8 connection(s)
I20260924 19:29:01.315474 483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:29:01.316380 551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:01.318585 551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f: Bootstrap starting.
I20260924 19:29:01.319139 551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:01.319412 551 log.cc:824] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:01.320067 551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f: No bootstrap required, opened a new log
I20260924 19:29:01.321648 551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36693 } }
I20260924 19:29:01.321781 551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:01.321822 551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6877d1d28d8d4efc944a845fbd77ab6f, State: Initialized, Role: FOLLOWER
I20260924 19:29:01.321987 551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36693 } }
I20260924 19:29:01.322072 551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:29:01.322121 551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:29:01.322173 551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:01.322690 551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36693 } }
I20260924 19:29:01.322800 551 leader_election.cc:302] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6877d1d28d8d4efc944a845fbd77ab6f; no voters:
I20260924 19:29:01.322978 551 leader_election.cc:288] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:29:01.323005 554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:29:01.323139 554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [term 1 LEADER]: Becoming Leader. State: Replica: 6877d1d28d8d4efc944a845fbd77ab6f, State: Running, Role: LEADER
I20260924 19:29:01.323225 551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:29:01.323314 554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36693 } }
I20260924 19:29:01.323657 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6877d1d28d8d4efc944a845fbd77ab6f. Latest consensus state: current_term: 1 leader_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36693 } } }
I20260924 19:29:01.323786 555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6877d1d28d8d4efc944a845fbd77ab6f" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36693 } } }
I20260924 19:29:01.323796 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:01.323886 555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:01.324314 564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:29:01.324776 564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:29:01.325158 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 483
I20260924 19:29:01.325265 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:29:01.325467 19350 external_mini_cluster.cc:1428] Setting key 26924e9082f83fc2aa71fc2ae22bcb07
I20260924 19:29:01.328558 564 catalog_manager.cc:1347] Generated new cluster ID: 620f848fb8684b1d9f072c5c7ad127e2
I20260924 19:29:01.328617 564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:29:01 dist-test-slave-r8cc krb5kdc[470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:01.337455 574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:01.325893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.18.229.190:36693)
0924 19:29:01.328567 (+ 2674us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:01.328575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:01.328607 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:29:01.328728 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:01.328741 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:01.328784 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:01.328932 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:01.329573 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:01.330464 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:01.331224 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:01.331635 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:01.331748 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:01.334176 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:01.334180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:01.334180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:01.334197 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:01.335872 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:01.336315 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:01.336318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:01.336318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:01.336412 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:01.336622 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:01.336623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:01.336623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:01.336847 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:01.337029 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:01.337106 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:29:01.337244 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2630,"thread_start_us":67,"threads_started":1}
I20260924 19:29:01.338270 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:29:01.354889 564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:29:01.355644 564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:29:01.374095 564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6877d1d28d8d4efc944a845fbd77ab6f: Generated new TSK 0
I20260924 19:29:01.374316 564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:29:01.392082 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 483
2026-09-24T19:29:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1330 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:29:01 dist-test-slave-r8cc krb5kdc[578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:29:01 dist-test-slave-r8cc krb5kdc[578](info): set up 2 sockets
Sep 24 19:29:01 dist-test-slave-r8cc krb5kdc[578](info): commencing operation
krb5kdc: starting...
W20260924 19:29:02.424093 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:29:02 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:29:02.538688 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43121
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:02.620044 591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:02.620332 591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:02.620371 591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:02.621548 591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:29:02.621608 591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:02.621632 591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:02.621662 591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:29:02.621687 591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:29:02.623090 591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:02.623350 591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:02.623559 591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:02.628193 597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:02.628254 599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:02.628274 591 server_base.cc:1034] running on GCE node
W20260924 19:29:02.628193 596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:02.628671 591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:02.628912 591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:02.630040 591 hybrid_clock.cc:648] HybridClock initialized: now 1790278142630031 us; error 28 us; skew 500 ppm
Sep 24 19:29:02 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:02.632416 591 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:29:02.632877 591 webserver.cc:460] Webserver started at http://127.18.229.190:46869/ using document root <none> and password file <none>
I20260924 19:29:02.633095 591 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:02.633136 591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:02.633277 591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:02.634294 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "261968c050d94390b88f3d4ef7455dd3"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "8da46d9d4a7ca4f410da1a63a1a1414f"
server_key_iv: "9bc5349af33ae4ab1213e418110e1642"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.634629 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "261968c050d94390b88f3d4ef7455dd3"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "8da46d9d4a7ca4f410da1a63a1a1414f"
server_key_iv: "9bc5349af33ae4ab1213e418110e1642"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.635875 591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:29:02.636617 606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.636768 591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.636861 591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "261968c050d94390b88f3d4ef7455dd3"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "8da46d9d4a7ca4f410da1a63a1a1414f"
server_key_iv: "9bc5349af33ae4ab1213e418110e1642"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.636940 591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:02.645558 591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:02.647223 591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:02.647375 591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:02.651098 591 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:43121
I20260924 19:29:02.651152 658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:43121 every 8 connection(s)
I20260924 19:29:02.651481 591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:29:02.652519 659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:02.653048 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 591
I20260924 19:29:02.653160 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:29:02.653355 19350 external_mini_cluster.cc:1428] Setting key a78e47b760568ede3af030498b8b6b65
I20260924 19:29:02.655265 659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap starting.
I20260924 19:29:02.655836 659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:02.656173 659 log.cc:824] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:02.656848 659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: No bootstrap required, opened a new log
I20260924 19:29:02.659101 659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:02.659296 659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:02.659333 659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261968c050d94390b88f3d4ef7455dd3, State: Initialized, Role: FOLLOWER
I20260924 19:29:02.659459 659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:02.659576 659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:29:02.659622 659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:29:02.659669 659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 19:29:02 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:02.660310 659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:02.660430 659 leader_election.cc:302] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261968c050d94390b88f3d4ef7455dd3; no voters:
I20260924 19:29:02.660606 659 leader_election.cc:288] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:29:02.660650 664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:29:02.660822 664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 261968c050d94390b88f3d4ef7455dd3, State: Running, Role: LEADER
I20260924 19:29:02.660876 659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:29:02.660959 664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:02.661262 664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261968c050d94390b88f3d4ef7455dd3. Latest consensus state: current_term: 1 leader_uuid: "261968c050d94390b88f3d4ef7455dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } } }
I20260924 19:29:02.661345 664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:02.661273 665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261968c050d94390b88f3d4ef7455dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } } }
I20260924 19:29:02.661706 665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:02.661830 669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:29:02.662287 669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:29:02.663789 669 catalog_manager.cc:1347] Generated new cluster ID: 7dbd265fc8b742e887a6660e20982abe
I20260924 19:29:02.663849 669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:29:02.664477 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:02.653853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.18.229.190:43121)
0924 19:29:02.654003 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:02.654010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:02.654036 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:29:02.654405 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:02.654411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:02.654454 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:02.654568 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:02.655307 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:02.656309 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:02.657190 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:02.657804 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:02.657917 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:02.660287 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:02.660290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:02.660291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:02.660303 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:02.662609 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:02.663008 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:02.663011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:02.663012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:02.663120 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:02.663333 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:02.663334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:02.663334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:02.663655 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:02.664192 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:02.664253 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:29:02.664348 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260924 19:29:02.685611 669 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:29:02.686173 669 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:29:02.689545 669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Generated new TSK 0
I20260924 19:29:02.689740 669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:29:02.722380 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43121
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:02.801780 685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:02.802016 685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:02.802062 685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:02.803215 685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:02.803280 685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:02.803325 685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:29:02.804778 685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:02.805027 685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:02.805255 685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:02.810173 693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:02.810128 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:02.810141 691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:02.810412 685 server_base.cc:1034] running on GCE node
I20260924 19:29:02.810654 685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:02.810887 685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:02.812048 685 hybrid_clock.cc:648] HybridClock initialized: now 1790278142812031 us; error 35 us; skew 500 ppm
Sep 24 19:29:02 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:02.814397 685 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:29:02.814860 685 webserver.cc:460] Webserver started at http://127.18.229.129:39219/ using document root <none> and password file <none>
I20260924 19:29:02.815095 685 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:02.815157 685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:02.815291 685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:02.816370 685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "bb433dec8341427083b5ea07e90a2137"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "03ba098fe7052dff1739ad71e7062bde"
server_key_iv: "942fbe43cf9738e44e3d7feaa39aa4ca"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.816752 685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "bb433dec8341427083b5ea07e90a2137"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "03ba098fe7052dff1739ad71e7062bde"
server_key_iv: "942fbe43cf9738e44e3d7feaa39aa4ca"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.818109 685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:29:02.818816 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.818991 685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:02.819096 685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "bb433dec8341427083b5ea07e90a2137"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "03ba098fe7052dff1739ad71e7062bde"
server_key_iv: "942fbe43cf9738e44e3d7feaa39aa4ca"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.819165 685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:02.840467 685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:02.842291 685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:02.842433 685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:02.842643 685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:02.842955 685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:02.843009 685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.843057 685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:02.843084 685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.848685 685 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:41263
I20260924 19:29:02.848726 813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:41263 every 8 connection(s)
I20260924 19:29:02.849025 685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:29:02 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:02.857321 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 685
I20260924 19:29:02.857446 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:29:02.857549 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:02.849657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:42873 (local address 127.18.229.190:43121)
0924 19:29:02.849720 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:02.849722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:02.849915 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:29:02.850389 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:02.850391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:02.850400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:02.850451 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:02.851355 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:02.851883 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:02.852891 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:02.853098 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:02.853153 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:02.855091 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:02.855093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:02.855093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:02.855100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:02.856367 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:02.856721 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:02.856721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:02.856722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:02.856772 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:02.856967 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:02.856967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:02.856968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:02.857067 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:02.857118 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:02.857351 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 19:29:02.857444 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:29:02.857661 19350 external_mini_cluster.cc:1428] Setting key 299023a5cd2f07d53d13875bcd2c01f4
I20260924 19:29:02.857710 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:02.849656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:42873)
0924 19:29:02.849809 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:02.849844 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:02.850306 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:02.850524 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:02.850526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:02.851092 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:02.851278 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:02.851283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:02.851969 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:02.851970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:02.852820 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:02.852823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:02.853168 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:02.853168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:02.853264 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:02.853696 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:02.853704 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:02.855006 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:02.856439 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:02.856445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:02.856447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:02.856658 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:02.856824 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:02.856825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:02.856825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:02.856917 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:02.857116 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:02.857120 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:02.857227 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:29:02.857308 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:29:02.857462 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260924 19:29:02.858281 814 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43121
I20260924 19:29:02.858390 814 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:02.858589 814 heartbeater.cc:511] Master 127.18.229.190:43121 requested a full tablet report, sending...
I20260924 19:29:02.859102 623 ts_manager.cc:194] Registered new tserver with Master: bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263)
I20260924 19:29:02.859702 623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:42873
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:29:02.882591 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43121
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:02.963804 819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:02.963994 819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:02.964016 819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:02.965165 819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:02.965205 819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:02.965231 819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:29:02.966809 819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:02.967044 819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:02.967314 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:02.972074 824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:02.972095 825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:02.972074 827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:02.972255 819 server_base.cc:1034] running on GCE node
I20260924 19:29:02.972512 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:02.972738 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:02.973878 819 hybrid_clock.cc:648] HybridClock initialized: now 1790278142973861 us; error 34 us; skew 500 ppm
Sep 24 19:29:02 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:02.976229 819 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:29:02.976650 819 webserver.cc:460] Webserver started at http://127.18.229.130:40419/ using document root <none> and password file <none>
I20260924 19:29:02.976897 819 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:02.976963 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:02.977110 819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:02.978125 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "dfe591806cb747f08a64a82097b98069"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "5e309e1432c2c31adde89cb2f005afbb"
server_key_iv: "feda6fa58770e3478424c203951a61dd"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.978485 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "dfe591806cb747f08a64a82097b98069"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "5e309e1432c2c31adde89cb2f005afbb"
server_key_iv: "feda6fa58770e3478424c203951a61dd"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.979765 819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:29:02.980538 834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.980697 819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:02.980788 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "dfe591806cb747f08a64a82097b98069"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "5e309e1432c2c31adde89cb2f005afbb"
server_key_iv: "feda6fa58770e3478424c203951a61dd"
server_key_version: "encryptionkey@0"
I20260924 19:29:02.980863 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:03.010414 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:03.012403 819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:03.012564 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:03.012804 819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:03.013120 819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:03.013170 819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.013211 819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:03.013242 819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.019075 819 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41319
I20260924 19:29:03.019148 947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41319 every 8 connection(s)
I20260924 19:29:03.019470 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278142, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:03.027483 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 819
I20260924 19:29:03.027590 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:29:03.027827 19350 external_mini_cluster.cc:1428] Setting key 741ab43e18e8e930f7c2b698da2f8591
I20260924 19:29:03.028759 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.020211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:43571 (local address 127.18.229.190:43121)
0924 19:29:03.020255 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.020257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.020502 (+ 245us) server_negotiation.cc:407] Connection header received
0924 19:29:03.021056 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.021057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.021071 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.021122 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.022166 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.022677 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.023706 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.023903 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.024002 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.026083 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.026084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.026085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.026092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.027283 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.027793 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.027794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.027794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.027847 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.028157 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.028158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.028159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.028279 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.028357 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.028630 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.028695 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:29:03.029040 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.020203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:43571)
0924 19:29:03.020376 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.020415 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.020940 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.021191 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.021195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.021856 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:03.022064 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.022069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.022794 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.022795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.023602 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.023604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.024010 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.024012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.024195 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.024649 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:03.024657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:03.026001 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:03.027362 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.027371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.027373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.027713 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.027938 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.027939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.027939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.028064 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.028347 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:03.028354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:03.028482 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.028596 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.028766 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260924 19:29:03.029506 948 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43121
I20260924 19:29:03.029614 948 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:03.029790 948 heartbeater.cc:511] Master 127.18.229.190:43121 requested a full tablet report, sending...
I20260924 19:29:03.030216 623 ts_manager.cc:194] Registered new tserver with Master: dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
I20260924 19:29:03.030586 623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:43571
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:29:03.051115 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43121
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:03.130311 953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:03.130494 953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:03.130517 953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:03.131577 953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:03.131613 953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:03.131639 953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:29:03.133033 953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:03.133210 953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:03.133374 953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:03.138180 958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.138216 959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.138190 961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.138228 953 server_base.cc:1034] running on GCE node
I20260924 19:29:03.138545 953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:03.138783 953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:03.139912 953 hybrid_clock.cc:648] HybridClock initialized: now 1790278143139902 us; error 26 us; skew 500 ppm
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:03.142436 953 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:29:03.142910 953 webserver.cc:460] Webserver started at http://127.18.229.131:41833/ using document root <none> and password file <none>
I20260924 19:29:03.143150 953 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:03.143195 953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:03.143339 953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:03.144457 953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "55f0e3e934cf440c932f795f629aca30"
format_stamp: "Formatted at 2026-09-24 19:29:03 on dist-test-slave-r8cc"
server_key: "679004e186103423d889c0a7707db3e5"
server_key_iv: "6ae7322493a8fafb65bd0f0ffa64592e"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.144814 953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "55f0e3e934cf440c932f795f629aca30"
format_stamp: "Formatted at 2026-09-24 19:29:03 on dist-test-slave-r8cc"
server_key: "679004e186103423d889c0a7707db3e5"
server_key_iv: "6ae7322493a8fafb65bd0f0ffa64592e"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.146083 953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:29:03.146781 968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.146946 953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.147024 953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "55f0e3e934cf440c932f795f629aca30"
format_stamp: "Formatted at 2026-09-24 19:29:03 on dist-test-slave-r8cc"
server_key: "679004e186103423d889c0a7707db3e5"
server_key_iv: "6ae7322493a8fafb65bd0f0ffa64592e"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.147095 953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:03.167584 953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:03.169948 953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:03.170073 953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:03.170306 953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:03.170626 953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:03.170660 953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.170701 953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:03.170730 953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.176656 953 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:39259
I20260924 19:29:03.176699 1081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:39259 every 8 connection(s)
I20260924 19:29:03.177047 953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:03.185602 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 953
I20260924 19:29:03.185714 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:29:03.185798 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.177758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:59183 (local address 127.18.229.190:43121)
0924 19:29:03.177810 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.177813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.178006 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:29:03.178482 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.178483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.178493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.178562 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.179465 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.179954 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.180973 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.181169 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.181223 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.183344 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.183345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.183346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.183354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.184614 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.184958 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.184959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.184960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.185007 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.185219 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.185220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.185220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.185314 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.185363 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.185600 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.185683 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:29:03.185912 19350 external_mini_cluster.cc:1428] Setting key 4dba2ecbac3a1e09f2a3ea8d5a5799cf
I20260924 19:29:03.185982 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.177725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:59183)
0924 19:29:03.177881 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.177923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.178406 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.178618 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.178621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.179188 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:03.179389 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.179393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.180045 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.180046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.180904 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.180907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.181245 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.181245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.181345 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.181877 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:03.181886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:03.183224 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:03.184681 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.184687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.184689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.184897 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.185071 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.185072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.185072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.185169 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.185363 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:03.185367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:03.185476 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.185558 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.185704 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260924 19:29:03.186474 1082 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43121
I20260924 19:29:03.186589 1082 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:03.186781 1082 heartbeater.cc:511] Master 127.18.229.190:43121 requested a full tablet report, sending...
I20260924 19:29:03.187150 623 ts_manager.cc:194] Registered new tserver with Master: 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259)
I20260924 19:29:03.187479 623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:59183
I20260924 19:29:03.197618 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:29:03.204330 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.198899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.18.229.190:43121)
0924 19:29:03.198956 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.198958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.199073 (+ 115us) server_negotiation.cc:407] Connection header received
0924 19:29:03.199115 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.199116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.199124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.199169 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.199690 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.200237 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.200904 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.201087 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.201135 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.202180 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.202181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.202181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.202187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.203319 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.203615 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.203616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.203616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.203659 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.203807 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.203808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.203808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.203940 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.204010 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.204154 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.204239 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:29:03.205715 623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:29:03.207327 623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:29:03.213613 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.209517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:41263 (local address 127.0.0.1:50824)
0924 19:29:03.209824 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.209833 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.209874 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.210245 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.210246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.210248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.210330 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.210333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.211303 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.211305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.212500 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.212503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.213327 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.213328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.213406 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.213419 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.213491 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.213541 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":21,"thread_start_us":93,"threads_started":1}
I20260924 19:29:03.213629 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.209624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.18.229.129:41263)
0924 19:29:03.210108 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.210112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.210119 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:29:03.210149 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.210151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.210162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.210248 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.210411 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.211209 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.212596 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.213244 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.213330 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.213388 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.213514 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.213555 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":92,"threads_started":1}
I20260924 19:29:03.213924 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.209562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41319 (local address 127.0.0.1:59450)
0924 19:29:03.209886 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.209896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.209948 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.210228 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.210231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.210233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.210320 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.210323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.211689 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.211690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.212736 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.212738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.213627 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.213628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.213749 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.213754 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.213810 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.213851 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":32,"thread_start_us":71,"threads_started":1}
I20260924 19:29:03.213963 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.209800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.18.229.130:41319)
0924 19:29:03.210078 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.210082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.210090 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:29:03.210120 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.210122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.210134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.210218 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.210505 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.211587 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.212812 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.213468 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.213542 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.213592 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.213840 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.213883 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":74,"threads_started":1}
I20260924 19:29:03.214092 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.209513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39259 (local address 127.0.0.1:39650)
0924 19:29:03.209680 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.209723 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.209797 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.210578 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.210579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.210581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.210631 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.210632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.211853 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.211854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.212913 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.212916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.213862 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.213863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.213929 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.213933 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.213981 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.214021 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":43,"threads_started":1}
I20260924 19:29:03.214378 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.209726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.18.229.131:39259)
0924 19:29:03.210386 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.210390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.210399 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:29:03.210433 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.210435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.210451 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.210517 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.210732 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.211762 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.212996 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.213713 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.213780 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.213832 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.214228 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.214304 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":78,"threads_started":1}
I20260924 19:29:03.214721 882 tablet_service.cc:1467] Processing CreateTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc (DEFAULT_TABLE table=test-table [id=7c7a2afe0eaa42dcb4af5a3046e250d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:03.215057 882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9fa2ac29584401b00bcf4b7077b3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:03.215034 1016 tablet_service.cc:1467] Processing CreateTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc (DEFAULT_TABLE table=test-table [id=7c7a2afe0eaa42dcb4af5a3046e250d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:03.215342 1016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9fa2ac29584401b00bcf4b7077b3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:03.216724 748 tablet_service.cc:1467] Processing CreateTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc (DEFAULT_TABLE table=test-table [id=7c7a2afe0eaa42dcb4af5a3046e250d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:03.217027 748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9fa2ac29584401b00bcf4b7077b3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:03.218241 1099 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Bootstrap starting.
I20260924 19:29:03.218556 1100 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Bootstrap starting.
I20260924 19:29:03.218868 1099 tablet_bootstrap.cc:654] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:03.219182 1100 tablet_bootstrap.cc:654] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:03.219415 1099 log.cc:824] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.219769 1100 log.cc:824] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.220172 1099 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: No bootstrap required, opened a new log
I20260924 19:29:03.220247 1099 ts_tablet_manager.cc:1397] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:03.220528 1103 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Bootstrap starting.
I20260924 19:29:03.220638 1100 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: No bootstrap required, opened a new log
I20260924 19:29:03.220705 1100 ts_tablet_manager.cc:1397] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:03.221181 1103 tablet_bootstrap.cc:654] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:03.221745 1103 log.cc:824] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.222563 1103 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: No bootstrap required, opened a new log
I20260924 19:29:03.222487 1100 raft_consensus.cc:348] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.222630 1103 ts_tablet_manager.cc:1397] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:03.222625 1100 raft_consensus.cc:374] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:03.222666 1100 raft_consensus.cc:729] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f0e3e934cf440c932f795f629aca30, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.222770 1100 consensus_queue.cc:260] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.222828 1099 raft_consensus.cc:348] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.222987 1099 raft_consensus.cc:374] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:03.223039 1099 raft_consensus.cc:729] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe591806cb747f08a64a82097b98069, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.223053 1100 ts_tablet_manager.cc:1428] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:03.223163 1082 heartbeater.cc:503] Master 127.18.229.190:43121 was elected leader, sending a full tablet report...
I20260924 19:29:03.223169 1099 consensus_queue.cc:260] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.223438 1099 ts_tablet_manager.cc:1428] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:29:03.223481 948 heartbeater.cc:503] Master 127.18.229.190:43121 was elected leader, sending a full tablet report...
I20260924 19:29:03.224625 1103 raft_consensus.cc:348] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.224740 1103 raft_consensus.cc:374] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:03.224759 1103 raft_consensus.cc:729] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb433dec8341427083b5ea07e90a2137, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.224833 1103 consensus_queue.cc:260] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.225065 814 heartbeater.cc:503] Master 127.18.229.190:43121 was elected leader, sending a full tablet report...
I20260924 19:29:03.225315 1103 ts_tablet_manager.cc:1428] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:29:03.263626 1106 raft_consensus.cc:482] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:03.263742 1106 raft_consensus.cc:504] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.264047 1106 leader_election.cc:288] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263)
I20260924 19:29:03.266832 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.264237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:41263 (local address 127.18.229.130:54579)
0924 19:29:03.264310 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.264324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.264365 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.264518 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.264519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.264520 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.264594 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.264596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.265205 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.265206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.265941 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.265942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.266533 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.266534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.266637 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.266641 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.266700 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.266745 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:29:03.266842 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.264304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:54579 (local address 127.18.229.129:41263)
0924 19:29:03.264348 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.264350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.264361 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:29:03.264439 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.264441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.264454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.264506 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.264661 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.265129 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.266031 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.266458 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.266537 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.266547 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.266720 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.266773 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:29:03.267067 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.264327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:39949 (local address 127.18.229.131:39259)
0924 19:29:03.264380 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.264382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.264512 (+ 130us) server_negotiation.cc:407] Connection header received
0924 19:29:03.264591 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.264592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.264601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.264668 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.264829 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.265292 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.266187 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.266648 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.266731 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.266739 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.266949 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.266994 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:29:03.267158 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9fa2ac29584401b00bcf4b7077b3fc" candidate_uuid: "dfe591806cb747f08a64a82097b98069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb433dec8341427083b5ea07e90a2137" is_pre_election: true
I20260924 19:29:03.267290 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.264238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39259 (local address 127.18.229.130:39949)
0924 19:29:03.264436 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.264444 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.264530 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.264673 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.264674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.264675 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.264758 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.264759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.265371 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.265372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.266100 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.266102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.266785 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.266787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.266859 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.266864 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.267179 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.267222 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":13,"thread_start_us":123,"threads_started":1}
I20260924 19:29:03.267305 768 raft_consensus.cc:2454] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe591806cb747f08a64a82097b98069 in term 0.
I20260924 19:29:03.267578 835 leader_election.cc:302] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb433dec8341427083b5ea07e90a2137, dfe591806cb747f08a64a82097b98069; no voters:
I20260924 19:29:03.267623 1036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9fa2ac29584401b00bcf4b7077b3fc" candidate_uuid: "dfe591806cb747f08a64a82097b98069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f0e3e934cf440c932f795f629aca30" is_pre_election: true
I20260924 19:29:03.267724 1106 raft_consensus.cc:2781] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:29:03.267752 1036 raft_consensus.cc:2454] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe591806cb747f08a64a82097b98069 in term 0.
I20260924 19:29:03.267793 1106 raft_consensus.cc:482] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:29:03.267818 1106 raft_consensus.cc:3037] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:03.268556 1106 raft_consensus.cc:504] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.268692 1106 leader_election.cc:288] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [CANDIDATE]: Term 1 election: Requested vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263)
I20260924 19:29:03.268823 1036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9fa2ac29584401b00bcf4b7077b3fc" candidate_uuid: "dfe591806cb747f08a64a82097b98069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f0e3e934cf440c932f795f629aca30"
I20260924 19:29:03.268833 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9fa2ac29584401b00bcf4b7077b3fc" candidate_uuid: "dfe591806cb747f08a64a82097b98069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb433dec8341427083b5ea07e90a2137"
I20260924 19:29:03.268911 768 raft_consensus.cc:3037] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:03.268915 1036 raft_consensus.cc:3037] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:03.269604 768 raft_consensus.cc:2454] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe591806cb747f08a64a82097b98069 in term 1.
I20260924 19:29:03.269701 1036 raft_consensus.cc:2454] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe591806cb747f08a64a82097b98069 in term 1.
I20260924 19:29:03.269778 835 leader_election.cc:302] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb433dec8341427083b5ea07e90a2137, dfe591806cb747f08a64a82097b98069; no voters:
I20260924 19:29:03.269887 1106 raft_consensus.cc:2781] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 19:29:03.269979 949 tablet.cc:2369] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:29:03.270048 1106 raft_consensus.cc:686] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 LEADER]: Becoming Leader. State: Replica: dfe591806cb747f08a64a82097b98069, State: Running, Role: LEADER
I20260924 19:29:03.270141 1106 consensus_queue.cc:237] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.270919 623 catalog_manager.cc:5495] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe591806cb747f08a64a82097b98069 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "dfe591806cb747f08a64a82097b98069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } health_report { overall_health: HEALTHY } } }
I20260924 19:29:03.285861 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.283707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.18.229.130:41319)
0924 19:29:03.283762 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.283764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.283775 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:29:03.283823 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.283823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.283839 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.283899 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:29:03.284020 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.284568 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.285264 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.285439 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.285483 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.285600 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:29:03.285692 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:29:03.285750 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.285778 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.285793 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:29:03.287463 1036 raft_consensus.cc:1261] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Refusing update from remote peer dfe591806cb747f08a64a82097b98069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:03.287483 768 raft_consensus.cc:1261] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Refusing update from remote peer dfe591806cb747f08a64a82097b98069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:03.287824 1109 consensus_queue.cc:1035] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:03.287916 1106 consensus_queue.cc:1035] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:03.290727 1113 mvcc.cc:205] Tried to move back new op lower bound from 7332979274902982656 to 7332979274835009536. Current Snapshot: MvccSnapshot[applied={T|T < 7332979274902982656}]
I20260924 19:29:03.290843 1114 mvcc.cc:205] Tried to move back new op lower bound from 7332979274902982656 to 7332979274835009536. Current Snapshot: MvccSnapshot[applied={T|T < 7332979274902982656 or (T in {7332979274902982656})}]
I20260924 19:29:03.292163 622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:03.292264 622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:03.293013 622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Sending DeleteTablet for 3 replicas of tablet bd9fa2ac29584401b00bcf4b7077b3fc
I20260924 19:29:03.293325 882 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:59450
I20260924 19:29:03.293349 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39650
I20260924 19:29:03.293361 748 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50824
I20260924 19:29:03.293614 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 685
I20260924 19:29:03.293648 1123 tablet_replica.cc:330] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: stopping tablet replica
I20260924 19:29:03.293736 1123 raft_consensus.cc:2229] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:03.293833 1123 raft_consensus.cc:2258] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:03.294191 1124 tablet_replica.cc:330] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: stopping tablet replica
I20260924 19:29:03.294288 1124 raft_consensus.cc:2229] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:03.294386 1124 raft_consensus.cc:2258] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:03.294833 1124 ts_tablet_manager.cc:1905] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:03.296774 1124 ts_tablet_manager.cc:1918] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:03.296844 1124 log.cc:1196] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/bd9fa2ac29584401b00bcf4b7077b3fc
I20260924 19:29:03.297116 1124 ts_tablet_manager.cc:1939] T bd9fa2ac29584401b00bcf4b7077b3fc P 55f0e3e934cf440c932f795f629aca30: Deleting consensus metadata
I20260924 19:29:03.297612 609 catalog_manager.cc:4841] TS 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): tablet bd9fa2ac29584401b00bcf4b7077b3fc (table test-table [id=7c7a2afe0eaa42dcb4af5a3046e250d1]) successfully deleted
I20260924 19:29:03.300283 1122 tablet_replica.cc:330] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: stopping tablet replica
I20260924 19:29:03.300400 1122 raft_consensus.cc:2229] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:29:03.300511 1122 raft_consensus.cc:2258] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:29:03.300876 607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:41263 (error 108)
I20260924 19:29:03.300935 1122 ts_tablet_manager.cc:1905] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:03.301128 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 819
I20260924 19:29:03.301501 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.301391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:41263 (local address 127.0.0.1:50834)
0924 19:29:03.301435 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.301464 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:41263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:29:03.301671 607 catalog_manager.cc:4584] TS bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd9fa2ac29584401b00bcf4b7077b3fc: Network error: Client connection negotiation failed: client connection to 127.18.229.129:41263: connect: Connection refused (error 111)
I20260924 19:29:03.305894 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.305796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41319 (local address 127.0.0.1:59468)
0924 19:29:03.305843 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.305867 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:29:03.305904 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 953
I20260924 19:29:03.310184 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 591
I20260924 19:29:03.315326 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43121
--webserver_interface=127.18.229.190
--webserver_port=46869
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:03.395740 1126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:03.395939 1126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:03.395961 1126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:03.397135 1126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:29:03.397174 1126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:03.397189 1126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:03.397238 1126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:29:03.397254 1126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:29:03.398784 1126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=46869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:03.399030 1126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:03.399247 1126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:03.407688 1133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.407722 1132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.407744 1135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.408006 1126 server_base.cc:1034] running on GCE node
I20260924 19:29:03.408257 1126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:03.408540 1126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:03.409670 1126 hybrid_clock.cc:648] HybridClock initialized: now 1790278143409660 us; error 29 us; skew 500 ppm
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:03.411960 1126 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:29:03.412395 1126 webserver.cc:460] Webserver started at http://127.18.229.190:46869/ using document root <none> and password file <none>
I20260924 19:29:03.412582 1126 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:03.412632 1126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:03.413964 1126 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:03.414564 1142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.414736 1126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:03.414817 1126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "261968c050d94390b88f3d4ef7455dd3"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "8da46d9d4a7ca4f410da1a63a1a1414f"
server_key_iv: "9bc5349af33ae4ab1213e418110e1642"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.415169 1126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:03.435663 1126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:03.437531 1126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:03.437641 1126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:03.441461 1126 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:43121
I20260924 19:29:03.441560 1194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:43121 every 8 connection(s)
I20260924 19:29:03.441890 1126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:29:03.443411 1195 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:29:03.444068 1195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap starting.
I20260924 19:29:03.449849 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1126
I20260924 19:29:03.450129 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:41263
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:43121
--webserver_port=39219
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:29:03.452245 1195 log.cc:824] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.454113 1195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:03.454406 1195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap complete.
I20260924 19:29:03.456688 1195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:03.456928 1195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261968c050d94390b88f3d4ef7455dd3, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.457116 1195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:03.457204 1195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:29:03.457261 1195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:29:03.457307 1195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:29:03.458263 1195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:03.458379 1195 leader_election.cc:302] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261968c050d94390b88f3d4ef7455dd3; no voters:
I20260924 19:29:03.458561 1195 leader_election.cc:288] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:29:03.458596 1199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:29:03.458703 1199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 261968c050d94390b88f3d4ef7455dd3, State: Running, Role: LEADER
I20260924 19:29:03.458828 1195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:29:03.458845 1199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:03.459282 1201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261968c050d94390b88f3d4ef7455dd3. Latest consensus state: current_term: 2 leader_uuid: "261968c050d94390b88f3d4ef7455dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } } }
I20260924 19:29:03.459290 1200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "261968c050d94390b88f3d4ef7455dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } } }
I20260924 19:29:03.459383 1201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:03.459398 1200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:03.460995 1215 catalog_manager.cc:1259] Loaded cluster ID: 7dbd265fc8b742e887a6660e20982abe
I20260924 19:29:03.461053 1215 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: loading cluster ID for follower catalog manager: success
W20260924 19:29:03.461256 1215 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 19:29:03.461306 1215 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 19:29:03.461376 1208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:29:03.461709 1208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:29:03.461807 1208 catalog_manager.cc:1259] Loaded cluster ID: 7dbd265fc8b742e887a6660e20982abe
I20260924 19:29:03.461838 1208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:29:03.462009 1208 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 19:29:03.464783 1208 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 19:29:03.466511 1208 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:29:03.466696 1208 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Loaded TSK: 0
I20260924 19:29:03.466889 1208 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:29:03.542593 1197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:03.542783 1197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:03.542805 1197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:03.543936 1197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:03.543975 1197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:03.544001 1197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:29:03.545434 1197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:41263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=39219
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:03.545666 1197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:03.545897 1197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:03.550736 1223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.550874 1197 server_base.cc:1034] running on GCE node
W20260924 19:29:03.550732 1225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.550763 1222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.551219 1197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:03.551489 1197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:03.552663 1197 hybrid_clock.cc:648] HybridClock initialized: now 1790278143552650 us; error 27 us; skew 500 ppm
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:03.555589 1197 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:29:03.556100 1197 webserver.cc:460] Webserver started at http://127.18.229.129:39219/ using document root <none> and password file <none>
I20260924 19:29:03.556387 1197 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:03.556471 1197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:03.558200 1197 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:29:03.558998 1232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.559180 1197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:03.559273 1197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "bb433dec8341427083b5ea07e90a2137"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "03ba098fe7052dff1739ad71e7062bde"
server_key_iv: "942fbe43cf9738e44e3d7feaa39aa4ca"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.559684 1197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:03.578874 1197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:03.581218 1197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:03.581367 1197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:03.581605 1197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:03.582072 1239 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:29:03.583994 1197 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:29:03.584040 1197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:29:03.584076 1197 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:29:03.585031 1197 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:29:03.585078 1197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:03.585259 1239 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Bootstrap starting.
I20260924 19:29:03.592504 1197 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:41263
I20260924 19:29:03.592564 1346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:41263 every 8 connection(s)
I20260924 19:29:03.593001 1197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:29:03.595156 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1197
I20260924 19:29:03.595476 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:41319
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:43121
--webserver_port=40419
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:29:03.598570 1239 log.cc:824] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.601927 1239 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:03.602283 1239 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Bootstrap complete.
I20260924 19:29:03.603598 1239 ts_tablet_manager.cc:1397] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20260924 19:29:03.605350 1239 raft_consensus.cc:348] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.605741 1239 raft_consensus.cc:729] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb433dec8341427083b5ea07e90a2137, State: Initialized, Role: FOLLOWER
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:03.606701 1239 consensus_queue.cc:260] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.607005 1239 ts_tablet_manager.cc:1428] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260924 19:29:03.612497 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.593846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:56623)
0924 19:29:03.594049 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.594095 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.594683 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.595317 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.595320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.595702 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:03.595961 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.595967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.597750 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.597751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.599116 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.599119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.599991 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.599992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.600166 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.600790 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:03.600800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:03.608200 (+ 7400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:03.610569 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.610594 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.610596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.611010 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.611291 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.611292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.611292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.611418 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.611831 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:03.611836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:03.611995 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.612129 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.612295 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260924 19:29:03.612519 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.593936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:56623 (local address 127.18.229.190:43121)
0924 19:29:03.594147 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.594157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.594201 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:29:03.594925 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.594934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.594993 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.595117 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.596153 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.597620 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.599214 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.599789 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.599860 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.608317 (+ 8457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.608321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.608321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.608334 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.610282 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.611102 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.611105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.611106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.611217 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.611483 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.611484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.611484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.611763 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.611842 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.612201 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.612337 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260924 19:29:03.613742 1347 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43121
I20260924 19:29:03.613842 1347 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:03.614019 1347 heartbeater.cc:511] Master 127.18.229.190:43121 requested a full tablet report, sending...
I20260924 19:29:03.614784 1159 ts_manager.cc:194] Registered new tserver with Master: bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263)
I20260924 19:29:03.615988 1159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:56623
I20260924 19:29:03.616952 1347 heartbeater.cc:503] Master 127.18.229.190:43121 was elected leader, sending a full tablet report...
I20260924 19:29:03.620404 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.615409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:41263 (local address 127.0.0.1:50846)
0924 19:29:03.617236 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.617264 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.617353 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.617583 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.617586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.617588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.617648 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.617650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.618452 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.618453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.619415 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.619418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.620067 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.620068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.620171 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.620185 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.620280 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.620336 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"thread_start_us":89,"threads_started":1}
I20260924 19:29:03.620404 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.615650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.18.229.129:41263)
0924 19:29:03.615809 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.615813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.617334 (+ 1521us) server_negotiation.cc:407] Connection header received
0924 19:29:03.617429 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.617431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.617449 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.617542 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.617733 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.618370 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.619507 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.619954 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.620067 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.620152 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.620297 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.620336 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260924 19:29:03.620783 1281 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50846
I20260924 19:29:03.621263 1356 tablet_replica.cc:330] stopping tablet replica
I20260924 19:29:03.621366 1356 raft_consensus.cc:2229] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:03.621429 1356 raft_consensus.cc:2258] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:03.621734 1356 ts_tablet_manager.cc:1905] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:03.624089 1356 ts_tablet_manager.cc:1918] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:03.624182 1356 log.cc:1196] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/bd9fa2ac29584401b00bcf4b7077b3fc
I20260924 19:29:03.624402 1356 ts_tablet_manager.cc:1939] T bd9fa2ac29584401b00bcf4b7077b3fc P bb433dec8341427083b5ea07e90a2137: Deleting consensus metadata
I20260924 19:29:03.624866 1143 catalog_manager.cc:4841] TS bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263): tablet bd9fa2ac29584401b00bcf4b7077b3fc (table test-table [id=7c7a2afe0eaa42dcb4af5a3046e250d1]) successfully deleted
W20260924 19:29:03.684250 1351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:03.684437 1351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:03.684458 1351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:03.685573 1351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:03.685612 1351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:03.685659 1351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:29:03.687002 1351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:41319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=40419
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:03.687176 1351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:03.687397 1351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:03.692260 1364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.692268 1361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.692296 1351 server_base.cc:1034] running on GCE node
W20260924 19:29:03.692270 1362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.692625 1351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:03.692870 1351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:03.694022 1351 hybrid_clock.cc:648] HybridClock initialized: now 1790278143694008 us; error 42 us; skew 500 ppm
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:03.696373 1351 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:29:03.696789 1351 webserver.cc:460] Webserver started at http://127.18.229.130:40419/ using document root <none> and password file <none>
I20260924 19:29:03.697023 1351 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:03.697104 1351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:03.698525 1351 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:03.699290 1371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.699456 1351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:03.699555 1351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "dfe591806cb747f08a64a82097b98069"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "5e309e1432c2c31adde89cb2f005afbb"
server_key_iv: "feda6fa58770e3478424c203951a61dd"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.699899 1351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:03.706521 1351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:03.708348 1351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:03.708482 1351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:03.708704 1351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:03.709103 1378 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:29:03.710428 1351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:29:03.710480 1351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:03.710527 1351 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:29:03.711125 1351 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:29:03.711175 1351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:03.711220 1378 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Bootstrap starting.
I20260924 19:29:03.712821 1378 log.cc:824] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.715804 1378 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:03.716182 1378 tablet_bootstrap.cc:492] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Bootstrap complete.
I20260924 19:29:03.716351 1378 ts_tablet_manager.cc:1397] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 19:29:03.717375 1351 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41319
I20260924 19:29:03.717413 1486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41319 every 8 connection(s)
I20260924 19:29:03.717795 1351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:29:03.717854 1378 raft_consensus.cc:348] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.718329 1378 raft_consensus.cc:729] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe591806cb747f08a64a82097b98069, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.718467 1378 consensus_queue.cc:260] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } }
I20260924 19:29:03.718755 1378 ts_tablet_manager.cc:1428] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:03.721621 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1351
I20260924 19:29:03.721908 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:39259
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:43121
--webserver_port=41833
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:03.726576 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.718493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:54635 (local address 127.18.229.190:43121)
0924 19:29:03.718541 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.718543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.718752 (+ 209us) server_negotiation.cc:407] Connection header received
0924 19:29:03.718913 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.718914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.718927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.718988 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.719644 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.720198 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.721167 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.721381 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.721464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.723906 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.723907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.723907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.723915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.725263 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.725678 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.725679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.725679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.725738 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.725965 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.725966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.725967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.726083 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.726155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.726402 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.726486 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:29:03.726747 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.718441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:54635)
0924 19:29:03.718619 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.718658 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.718835 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.719040 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.719043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.719368 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:03.719550 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.719554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.720301 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.720302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.721068 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.721070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.721472 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.721473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.721594 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.722137 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:03.722147 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:03.723816 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:03.725361 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.725367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.725369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.725592 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.725801 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.725802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.725802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.725901 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.726146 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:03.726152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:03.726281 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.726381 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.726537 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260924 19:29:03.727201 1487 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43121
I20260924 19:29:03.727301 1487 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:03.727484 1487 heartbeater.cc:511] Master 127.18.229.190:43121 requested a full tablet report, sending...
I20260924 19:29:03.728024 1159 ts_manager.cc:194] Registered new tserver with Master: dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
I20260924 19:29:03.728624 1159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:54635
I20260924 19:29:03.729503 1487 heartbeater.cc:503] Master 127.18.229.190:43121 was elected leader, sending a full tablet report...
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.130@KRBTEST.COM
I20260924 19:29:03.735572 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.728406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.18.229.130:41319)
0924 19:29:03.728627 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.728632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.728641 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:29:03.728674 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.728677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.728693 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.728761 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.729186 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.730029 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.730661 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.730967 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.731043 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.733029 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.733037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.733037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.733043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.734412 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.734827 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.734828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.734829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.734877 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.735088 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.735088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.735089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.735222 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.735283 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.735438 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.735500 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
I20260924 19:29:03.735597 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.728335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41319 (local address 127.0.0.1:59470)
0924 19:29:03.728435 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.728448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.728501 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.728776 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.728777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.729077 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:03.729108 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.729110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.730088 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.730089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.730574 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.730575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.731046 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.731047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.731104 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.731568 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:03.731577 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:03.732950 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:03.734484 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.734492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.734494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.734754 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.734935 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.734936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.734937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.735001 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.735281 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:03.735287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:03.735347 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.735418 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.735506 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 19:29:03.735957 1421 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9fa2ac29584401b00bcf4b7077b3fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:59470
I20260924 19:29:03.736626 1493 tablet_replica.cc:330] stopping tablet replica
I20260924 19:29:03.736730 1493 raft_consensus.cc:2229] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:03.736806 1493 raft_consensus.cc:2258] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:03.737056 1493 ts_tablet_manager.cc:1905] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:03.739073 1493 ts_tablet_manager.cc:1918] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:03.739161 1493 log.cc:1196] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/bd9fa2ac29584401b00bcf4b7077b3fc
I20260924 19:29:03.739372 1493 ts_tablet_manager.cc:1939] T bd9fa2ac29584401b00bcf4b7077b3fc P dfe591806cb747f08a64a82097b98069: Deleting consensus metadata
I20260924 19:29:03.739825 1144 catalog_manager.cc:4841] TS dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): tablet bd9fa2ac29584401b00bcf4b7077b3fc (table test-table [id=7c7a2afe0eaa42dcb4af5a3046e250d1]) successfully deleted
W20260924 19:29:03.803161 1491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:03.803408 1491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:03.803459 1491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:03.804612 1491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:03.804649 1491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:03.804697 1491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:29:03.806052 1491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:39259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=41833
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:03.806277 1491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:03.806489 1491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:03.811179 1501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.811285 1491 server_base.cc:1034] running on GCE node
W20260924 19:29:03.811209 1499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:03.811357 1498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:03.811589 1491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:03.811779 1491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:03.812929 1491 hybrid_clock.cc:648] HybridClock initialized: now 1790278143812920 us; error 35 us; skew 500 ppm
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:03.815215 1491 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:29:03.815639 1491 webserver.cc:460] Webserver started at http://127.18.229.131:41833/ using document root <none> and password file <none>
I20260924 19:29:03.815855 1491 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:03.815925 1491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:03.817302 1491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:03.817871 1508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.818025 1491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:03.818120 1491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "55f0e3e934cf440c932f795f629aca30"
format_stamp: "Formatted at 2026-09-24 19:29:03 on dist-test-slave-r8cc"
server_key: "679004e186103423d889c0a7707db3e5"
server_key_iv: "6ae7322493a8fafb65bd0f0ffa64592e"
server_key_version: "encryptionkey@0"
I20260924 19:29:03.818449 1491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:03.831207 1491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:03.833418 1491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:03.833565 1491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:03.833766 1491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:03.834059 1491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:03.834107 1491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.834154 1491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:03.834187 1491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:03.841210 1491 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:39259
I20260924 19:29:03.841296 1621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:39259 every 8 connection(s)
I20260924 19:29:03.841660 1491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:29:03.846403 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1491
Sep 24 19:29:03 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278143, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:03.851807 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.842428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:60417 (local address 127.18.229.190:43121)
0924 19:29:03.842478 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.842481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.842726 (+ 245us) server_negotiation.cc:407] Connection header received
0924 19:29:03.843476 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.843479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.843493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.843566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.844822 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.845312 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.846230 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.846409 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.846457 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.848874 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.848876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.848876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.848884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.850386 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.850812 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.850814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.850814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.850870 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.851161 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.851162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.851163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.851263 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.851315 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.851652 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.851745 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:29:03.852029 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.842422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:60417)
0924 19:29:03.842589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.842635 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.843324 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.843668 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.843673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.844449 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:03.844702 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.844708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.845381 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.845381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.846161 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.846162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.846485 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.846487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.846644 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.847232 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:03.847242 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:03.848795 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:03.850483 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.850489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.850492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.850728 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.850956 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:03.850956 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:03.850957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:03.851090 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:03.851327 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:03.851333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:03.851502 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.851628 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.851799 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260924 19:29:03.852512 1622 heartbeater.cc:347] Connected to a master server at 127.18.229.190:43121
I20260924 19:29:03.852612 1622 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:03.852794 1622 heartbeater.cc:511] Master 127.18.229.190:43121 requested a full tablet report, sending...
I20260924 19:29:03.853152 1159 ts_manager.cc:194] Registered new tserver with Master: 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259)
I20260924 19:29:03.853513 1159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:60417
I20260924 19:29:03.853577 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.846780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.18.229.190:43121)
0924 19:29:03.847023 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.847025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.847032 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:29:03.847063 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.847064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.847076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.847136 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:03.847841 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.848360 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.849284 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.849478 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.849561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.850698 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:03.850699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:03.850700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:03.850706 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:03.852433 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.852820 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.852821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.852821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.852878 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:03.853054 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:03.853055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:03.853055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:03.853159 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:03.853329 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.853366 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.853444 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":53,"threads_started":1}
I20260924 19:29:03.854315 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:29:03.856989 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.854625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.18.229.190:43121)
0924 19:29:03.854672 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.854673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.854764 (+ 91us) server_negotiation.cc:407] Connection header received
0924 19:29:03.854848 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.854849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.854885 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.854960 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:29:03.855067 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.855496 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.856244 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.856452 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.856530 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.856697 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:29:03.856794 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:29:03.856879 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.856898 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.856920 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:29:03.857254 1159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:29:03.858767 1159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:29:03.860769 1421 tablet_service.cc:1467] Processing CreateTablet for tablet 48786404ed884cd699f904bcf54bfff5 (DEFAULT_TABLE table=test-table [id=ee4618b1d34d4d08978109833bb9ada0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:03.860895 1281 tablet_service.cc:1467] Processing CreateTablet for tablet 48786404ed884cd699f904bcf54bfff5 (DEFAULT_TABLE table=test-table [id=ee4618b1d34d4d08978109833bb9ada0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:03.860954 1421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48786404ed884cd699f904bcf54bfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:03.861032 1281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48786404ed884cd699f904bcf54bfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:03.862502 1378 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Bootstrap starting.
I20260924 19:29:03.862928 1378 tablet_bootstrap.cc:654] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:03.863282 1239 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Bootstrap starting.
I20260924 19:29:03.863579 1239 tablet_bootstrap.cc:654] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:03.863807 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.860532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39259 (local address 127.0.0.1:39654)
0924 19:29:03.860573 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.860584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.860617 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.860924 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.860925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.860927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.860985 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.860986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.861904 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.861905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.862690 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.862694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.863532 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.863541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.863611 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.863615 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.863681 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.863725 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:29:03.863942 1378 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: No bootstrap required, opened a new log
I20260924 19:29:03.863987 1378 ts_tablet_manager.cc:1397] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:29:03.864053 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.860645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.18.229.131:39259)
0924 19:29:03.860792 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.860796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.860803 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:29:03.860832 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.860834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.860845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.860928 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.861045 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.861816 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.862774 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.863425 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.863529 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.863591 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.863911 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.863953 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260924 19:29:03.864320 1378 raft_consensus.cc:348] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.864393 1378 raft_consensus.cc:374] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:03.864446 1378 raft_consensus.cc:729] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe591806cb747f08a64a82097b98069, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.864500 1378 consensus_queue.cc:260] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.864596 1378 ts_tablet_manager.cc:1428] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:03.864892 1239 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: No bootstrap required, opened a new log
I20260924 19:29:03.864943 1239 ts_tablet_manager.cc:1397] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:29:03.865062 1239 raft_consensus.cc:348] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.865135 1239 raft_consensus.cc:374] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:03.865185 1239 raft_consensus.cc:729] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb433dec8341427083b5ea07e90a2137, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.865244 1239 consensus_queue.cc:260] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.865340 1239 ts_tablet_manager.cc:1428] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:03.865283 1556 tablet_service.cc:1467] Processing CreateTablet for tablet 48786404ed884cd699f904bcf54bfff5 (DEFAULT_TABLE table=test-table [id=ee4618b1d34d4d08978109833bb9ada0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:03.865612 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48786404ed884cd699f904bcf54bfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:03.868221 1631 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Bootstrap starting.
I20260924 19:29:03.868851 1631 tablet_bootstrap.cc:654] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:03.869316 1631 log.cc:824] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:03.869949 1631 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: No bootstrap required, opened a new log
I20260924 19:29:03.870002 1631 ts_tablet_manager.cc:1397] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:29:03.871276 1631 raft_consensus.cc:348] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.871397 1631 raft_consensus.cc:374] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:03.871446 1631 raft_consensus.cc:729] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f0e3e934cf440c932f795f629aca30, State: Initialized, Role: FOLLOWER
I20260924 19:29:03.871568 1631 consensus_queue.cc:260] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.871824 1631 ts_tablet_manager.cc:1428] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:03.871852 1622 heartbeater.cc:503] Master 127.18.229.190:43121 was elected leader, sending a full tablet report...
I20260924 19:29:03.926090 1633 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:03.926204 1633 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.926594 1633 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319), bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263)
I20260924 19:29:03.930433 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.926845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:53937 (local address 127.18.229.129:41263)
0924 19:29:03.926893 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.926895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.927026 (+ 131us) server_negotiation.cc:407] Connection header received
0924 19:29:03.927067 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.927069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.927081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.927137 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.927291 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.928039 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.929562 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.930090 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.930170 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.930180 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.930323 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.930367 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:29:03.930457 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.926852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:50653 (local address 127.18.229.130:41319)
0924 19:29:03.926893 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.926895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.926905 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:29:03.926967 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.926967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.926987 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.927094 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:03.927298 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.928047 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.929563 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.930167 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.930231 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.930310 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.930353 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.930387 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:29:03.930444 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.926770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41319 (local address 127.18.229.131:50653)
0924 19:29:03.926852 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.926863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.926909 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.927127 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.927128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.927130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.927220 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.927221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.928138 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.928138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.929468 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.929472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.930236 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.930237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.930305 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.930308 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.930351 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.930379 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":98}
I20260924 19:29:03.930454 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.926771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:41263 (local address 127.18.229.131:53937)
0924 19:29:03.926962 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.926970 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:03.927023 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:03.927144 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:03.927145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:03.927146 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:03.927220 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.927221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.928149 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.928150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.929468 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:03.929472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.930166 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.930167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:03.930245 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.930249 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:29:03.930303 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:29:03.930354 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":82,"thread_start_us":94,"threads_started":1}
I20260924 19:29:03.930781 1441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48786404ed884cd699f904bcf54bfff5" candidate_uuid: "55f0e3e934cf440c932f795f629aca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe591806cb747f08a64a82097b98069" is_pre_election: true
I20260924 19:29:03.930951 1441 raft_consensus.cc:2454] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55f0e3e934cf440c932f795f629aca30 in term 0.
I20260924 19:29:03.930974 1301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48786404ed884cd699f904bcf54bfff5" candidate_uuid: "55f0e3e934cf440c932f795f629aca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb433dec8341427083b5ea07e90a2137" is_pre_election: true
I20260924 19:29:03.931136 1301 raft_consensus.cc:2454] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55f0e3e934cf440c932f795f629aca30 in term 0.
I20260924 19:29:03.931205 1510 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069; no voters:
I20260924 19:29:03.931327 1633 raft_consensus.cc:2781] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:29:03.931381 1633 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:29:03.931413 1633 raft_consensus.cc:3037] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:03.932366 1633 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.932540 1633 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [CANDIDATE]: Term 1 election: Requested vote from peers dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319), bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263)
I20260924 19:29:03.932698 1301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48786404ed884cd699f904bcf54bfff5" candidate_uuid: "55f0e3e934cf440c932f795f629aca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb433dec8341427083b5ea07e90a2137"
I20260924 19:29:03.932686 1441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48786404ed884cd699f904bcf54bfff5" candidate_uuid: "55f0e3e934cf440c932f795f629aca30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfe591806cb747f08a64a82097b98069"
I20260924 19:29:03.932770 1441 raft_consensus.cc:3037] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:03.932770 1301 raft_consensus.cc:3037] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:03.933622 1441 raft_consensus.cc:2454] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55f0e3e934cf440c932f795f629aca30 in term 1.
I20260924 19:29:03.933671 1301 raft_consensus.cc:2454] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55f0e3e934cf440c932f795f629aca30 in term 1.
I20260924 19:29:03.933784 1510 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069; no voters:
I20260924 19:29:03.933895 1633 raft_consensus.cc:2781] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:29:03.934027 1633 raft_consensus.cc:686] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 1 LEADER]: Becoming Leader. State: Replica: 55f0e3e934cf440c932f795f629aca30, State: Running, Role: LEADER
I20260924 19:29:03.934129 1633 consensus_queue.cc:237] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:03.934935 1159 catalog_manager.cc:5495] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55f0e3e934cf440c932f795f629aca30 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "55f0e3e934cf440c932f795f629aca30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } health_report { overall_health: UNKNOWN } } }
I20260924 19:29:03.956058 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.954018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.18.229.131:39259)
0924 19:29:03.954073 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:03.954075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:03.954083 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:29:03.954159 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:03.954160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:03.954177 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:03.954236 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:29:03.954338 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.954788 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.955443 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:03.955616 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:03.955662 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:03.955773 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:29:03.955862 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:29:03.955917 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:03.955946 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:29:03.955961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:29:03.957741 1301 raft_consensus.cc:1261] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Refusing update from remote peer 55f0e3e934cf440c932f795f629aca30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:03.957746 1441 raft_consensus.cc:1261] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Refusing update from remote peer 55f0e3e934cf440c932f795f629aca30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:03.958065 1633 consensus_queue.cc:1035] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:03.958204 1635 consensus_queue.cc:1035] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:03.959990 1638 mvcc.cc:205] Tried to move back new op lower bound from 7332979277648568320 to 7332979277554974720. Current Snapshot: MvccSnapshot[applied={T|T < 7332979277648568320}]
I20260924 19:29:03.960355 1641 mvcc.cc:205] Tried to move back new op lower bound from 7332979277648568320 to 7332979277554974720. Current Snapshot: MvccSnapshot[applied={T|T < 7332979277648568320}]
I20260924 19:29:03.960543 1639 mvcc.cc:205] Tried to move back new op lower bound from 7332979277648568320 to 7332979277554974720. Current Snapshot: MvccSnapshot[applied={T|T < 7332979277648568320}]
I20260924 19:29:03.962673 1159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:03.962766 1159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:03.963464 1159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Sending DeleteTablet for 3 replicas of tablet 48786404ed884cd699f904bcf54bfff5
I20260924 19:29:03.963742 1556 tablet_service.cc:1514] Processing DeleteTablet for tablet 48786404ed884cd699f904bcf54bfff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:39654
I20260924 19:29:03.963790 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 1197
I20260924 19:29:03.963797 1421 tablet_service.cc:1514] Processing DeleteTablet for tablet 48786404ed884cd699f904bcf54bfff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:03 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:59470
I20260924 19:29:03.964272 1493 tablet_replica.cc:330] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: stopping tablet replica
I20260924 19:29:03.964329 1493 raft_consensus.cc:2229] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:03.964351 1647 tablet_replica.cc:330] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: stopping tablet replica
I20260924 19:29:03.964373 1493 raft_consensus.cc:2258] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:03.964443 1647 raft_consensus.cc:2229] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:29:03.964564 1647 raft_consensus.cc:2258] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:03.964721 1493 ts_tablet_manager.cc:1905] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:03.965503 1647 ts_tablet_manager.cc:1905] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:03.966143 1493 ts_tablet_manager.cc:1918] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:03.966195 1493 log.cc:1196] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/48786404ed884cd699f904bcf54bfff5
I20260924 19:29:03.966418 1493 ts_tablet_manager.cc:1939] T 48786404ed884cd699f904bcf54bfff5 P dfe591806cb747f08a64a82097b98069: Deleting consensus metadata
I20260924 19:29:03.966720 1144 catalog_manager.cc:4841] TS dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): tablet 48786404ed884cd699f904bcf54bfff5 (table test-table [id=ee4618b1d34d4d08978109833bb9ada0]) successfully deleted
I20260924 19:29:03.967190 1647 ts_tablet_manager.cc:1918] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:03.967252 1647 log.cc:1196] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/48786404ed884cd699f904bcf54bfff5
I20260924 19:29:03.967516 1647 ts_tablet_manager.cc:1939] T 48786404ed884cd699f904bcf54bfff5 P 55f0e3e934cf440c932f795f629aca30: Deleting consensus metadata
I20260924 19:29:03.967908 1145 catalog_manager.cc:4841] TS 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): tablet 48786404ed884cd699f904bcf54bfff5 (table test-table [id=ee4618b1d34d4d08978109833bb9ada0]) successfully deleted
W20260924 19:29:03.969715 1143 connection.cc:535] client connection to 127.18.229.129:41263 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:29:03.969805 1143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:29:03.970142 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 1351
I20260924 19:29:03.970264 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:03.970134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:41263 (local address 127.0.0.1:50858)
0924 19:29:03.970192 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:03.970231 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:41263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 19:29:03.971290 1143 catalog_manager.cc:4584] TS bb433dec8341427083b5ea07e90a2137 (127.18.229.129:41263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48786404ed884cd699f904bcf54bfff5: Network error: Client connection negotiation failed: client connection to 127.18.229.129:41263: connect: Connection refused (error 111)
I20260924 19:29:03.975258 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 1491
I20260924 19:29:03.979907 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 1126
I20260924 19:29:03.985009 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43121
--webserver_interface=127.18.229.190
--webserver_port=46869
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:04.064854 1649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:04.065099 1649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:04.065133 1649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:04.066321 1649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:29:04.066398 1649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:04.066426 1649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:04.066454 1649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:29:04.066478 1649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:29:04.067929 1649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=46869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:04.068210 1649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:04.068421 1649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:04.073446 1654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.073458 1657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.073472 1655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:04.073485 1649 server_base.cc:1034] running on GCE node
I20260924 19:29:04.073923 1649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:04.074266 1649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:04.075436 1649 hybrid_clock.cc:648] HybridClock initialized: now 1790278144075447 us; error 48 us; skew 500 ppm
Sep 24 19:29:04 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278144, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:04.077759 1649 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:29:04.078161 1649 webserver.cc:460] Webserver started at http://127.18.229.190:46869/ using document root <none> and password file <none>
I20260924 19:29:04.078368 1649 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:04.078433 1649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:04.079698 1649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:04.080353 1664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.080526 1649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:04.080623 1649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "261968c050d94390b88f3d4ef7455dd3"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "8da46d9d4a7ca4f410da1a63a1a1414f"
server_key_iv: "9bc5349af33ae4ab1213e418110e1642"
server_key_version: "encryptionkey@0"
I20260924 19:29:04.080956 1649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:04.100970 1649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:04.102754 1649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:04.102903 1649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:04.106686 1649 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:43121
I20260924 19:29:04.106732 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:43121 every 8 connection(s)
I20260924 19:29:04.107107 1649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:29:04.108717 1717 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:29:04.109335 1717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap starting.
I20260924 19:29:04.109562 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1649
I20260924 19:29:04.109874 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:41263
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:43121
--webserver_port=39219
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:29:04.118083 1717 log.cc:824] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:04.120324 1717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:04.120595 1717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3: Bootstrap complete.
I20260924 19:29:04.122749 1717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:04.122973 1717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261968c050d94390b88f3d4ef7455dd3, State: Initialized, Role: FOLLOWER
I20260924 19:29:04.123157 1717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:04.123246 1717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:29:04.123297 1717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:29:04.123343 1717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:29:04.124723 1717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:04.124850 1717 leader_election.cc:302] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261968c050d94390b88f3d4ef7455dd3; no voters:
I20260924 19:29:04.125026 1717 leader_election.cc:288] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:29:04.125084 1721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:29:04.125227 1721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [term 3 LEADER]: Becoming Leader. State: Replica: 261968c050d94390b88f3d4ef7455dd3, State: Running, Role: LEADER
I20260924 19:29:04.125281 1717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:29:04.125319 1721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } }
I20260924 19:29:04.125571 1722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "261968c050d94390b88f3d4ef7455dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } } }
I20260924 19:29:04.125658 1722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:04.125569 1721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261968c050d94390b88f3d4ef7455dd3. Latest consensus state: current_term: 3 leader_uuid: "261968c050d94390b88f3d4ef7455dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261968c050d94390b88f3d4ef7455dd3" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 43121 } } }
I20260924 19:29:04.125744 1721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261968c050d94390b88f3d4ef7455dd3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:04.126210 1725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:29:04.127066 1725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:29:04.127182 1725 catalog_manager.cc:1259] Loaded cluster ID: 7dbd265fc8b742e887a6660e20982abe
I20260924 19:29:04.127219 1725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 19:29:04.127753 1725 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb6addd9ad google::LogMessage::Fail() at ??:0
@ 0x7fdb6ade088a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb6addd4de google::LogMessage::Flush() at ??:0
@ 0x7fdb6addea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb72cfe0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdb72d078b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb6ccab0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb6cca0954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb6db046db start_thread at ??:0
@ 0x7fdb698db61f clone at ??:0
W20260924 19:29:04.226173 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:04.226441 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:04.226477 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:04.228197 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:04.228253 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:04.228294 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:29:04.230466 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:41263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=39219
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:04.230770 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:04.231051 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:04.236368 1756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.236373 1753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.236505 1754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:04.236867 1719 server_base.cc:1034] running on GCE node
I20260924 19:29:04.237051 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:04.237268 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:04.238420 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790278144238410 us; error 32 us; skew 500 ppm
Sep 24 19:29:04 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278144, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:04.240923 1719 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:29:04.241334 1719 webserver.cc:460] Webserver started at http://127.18.229.129:39219/ using document root <none> and password file <none>
I20260924 19:29:04.241572 1719 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:04.241638 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:04.243050 1719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:29:04.243777 1763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.243954 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:29:04.244056 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "bb433dec8341427083b5ea07e90a2137"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "03ba098fe7052dff1739ad71e7062bde"
server_key_iv: "942fbe43cf9738e44e3d7feaa39aa4ca"
server_key_version: "encryptionkey@0"
I20260924 19:29:04.244431 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:04.272179 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:04.274569 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:04.274727 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:04.274966 1719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:04.275413 1770 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:29:04.277382 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:29:04.277441 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:29:04.277482 1719 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:29:04.278110 1719 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:29:04.278159 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:04.278225 1770 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Bootstrap starting.
I20260924 19:29:04.286024 1719 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:41263
I20260924 19:29:04.286144 1877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:41263 every 8 connection(s)
I20260924 19:29:04.286450 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:29:04.289721 1770 log.cc:824] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:04.291788 1770 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:04.292263 1770 tablet_bootstrap.cc:492] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Bootstrap complete.
I20260924 19:29:04.293195 1770 ts_tablet_manager.cc:1397] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20260924 19:29:04.294221 1770 raft_consensus.cc:348] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:04.294477 1770 raft_consensus.cc:729] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb433dec8341427083b5ea07e90a2137, State: Initialized, Role: FOLLOWER
I20260924 19:29:04.294638 1770 consensus_queue.cc:260] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:04.294915 1770 ts_tablet_manager.cc:1428] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 19:29:04.295905 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1719
I20260924 19:29:04.296247 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:41319
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:43121
--webserver_port=40419
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:04.373807 1880 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:29:04.287214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:41549)
0924 19:29:04.287390 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.287440 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:04.288011 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:04.373604 (+ 85593us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
W20260924 19:29:04.374037 1878 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43121 (0 consecutive failures): Network error: Failed to ping master at 127.18.229.190:43121: Client connection negotiation failed: client connection to 127.18.229.190:43121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:29:04.374387 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.374260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:37427)
0924 19:29:04.374318 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.374365 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:29:04.374768 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.374682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:48459)
0924 19:29:04.374730 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.374753 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 19:29:04.374892 1878 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:29:04.397411 1883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:04.397611 1883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:04.397635 1883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:04.398828 1883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:04.398865 1883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:04.398918 1883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:29:04.400434 1883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:41319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=40419
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:04.400660 1883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:04.400882 1883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:04.405701 1889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.405712 1891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.405712 1888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:04.405841 1883 server_base.cc:1034] running on GCE node
I20260924 19:29:04.406118 1883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:04.406366 1883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:04.407568 1883 hybrid_clock.cc:648] HybridClock initialized: now 1790278144407560 us; error 36 us; skew 500 ppm
Sep 24 19:29:04 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278144, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:04.410012 1883 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:29:04.410466 1883 webserver.cc:460] Webserver started at http://127.18.229.130:40419/ using document root <none> and password file <none>
I20260924 19:29:04.410708 1883 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:04.410753 1883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:04.412057 1883 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:04.412681 1898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.412860 1883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.412925 1883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "dfe591806cb747f08a64a82097b98069"
format_stamp: "Formatted at 2026-09-24 19:29:02 on dist-test-slave-r8cc"
server_key: "5e309e1432c2c31adde89cb2f005afbb"
server_key_iv: "feda6fa58770e3478424c203951a61dd"
server_key_version: "encryptionkey@0"
I20260924 19:29:04.413269 1883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:04.426188 1883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:04.427819 1883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:04.427966 1883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:04.428246 1883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:04.428550 1883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:04.428583 1883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.428638 1883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:04.428668 1883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.434437 1883 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41319
I20260924 19:29:04.434485 2011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41319 every 8 connection(s)
I20260924 19:29:04.434804 1883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:29:04.435775 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.435375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:39861)
0924 19:29:04.435526 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.435629 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260924 19:29:04.436233 2012 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43121 (0 consecutive failures): Network error: Failed to ping master at 127.18.229.190:43121: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
I20260924 19:29:04.436487 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.436401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:60085)
0924 19:29:04.436446 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.436471 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:29:04.436710 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.436662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:39931)
0924 19:29:04.436682 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.436699 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260924 19:29:04.436810 2012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:29:04.441865 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 1883
I20260924 19:29:04.442114 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:39259
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:43121
--webserver_port=41833
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:04.521703 2015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:04.521903 2015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:04.521925 2015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:04.523078 2015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:04.523119 2015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:04.523168 2015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:29:04.524606 2015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:39259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=41833
--tserver_master_addrs=127.18.229.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:04.524842 2015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:04.525076 2015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:29:04.530062 2015 server_base.cc:1034] running on GCE node
W20260924 19:29:04.529983 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.529990 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:04.529982 2023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:04.530527 2015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:04.530766 2015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:04.531939 2015 hybrid_clock.cc:648] HybridClock initialized: now 1790278144531920 us; error 32 us; skew 500 ppm
Sep 24 19:29:04 dist-test-slave-r8cc krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278144, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:04.534375 2015 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:29:04.534754 2015 webserver.cc:460] Webserver started at http://127.18.229.131:41833/ using document root <none> and password file <none>
I20260924 19:29:04.534927 2015 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:04.534967 2015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:04.536238 2015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:29:04.536813 2030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.536963 2015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.537061 2015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "55f0e3e934cf440c932f795f629aca30"
format_stamp: "Formatted at 2026-09-24 19:29:03 on dist-test-slave-r8cc"
server_key: "679004e186103423d889c0a7707db3e5"
server_key_iv: "6ae7322493a8fafb65bd0f0ffa64592e"
server_key_version: "encryptionkey@0"
I20260924 19:29:04.537483 2015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:04.556087 2015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:04.558377 2015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:04.558482 2015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:04.558663 2015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:04.558969 2015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:04.559002 2015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.559054 2015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:04.559082 2015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:04.564592 2015 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:39259
I20260924 19:29:04.564653 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:39259 every 8 connection(s)
I20260924 19:29:04.564999 2015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:29:04.566056 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.565693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:45785)
0924 19:29:04.565839 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.565915 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260924 19:29:04.566453 2144 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:43121 (0 consecutive failures): Network error: Failed to ping master at 127.18.229.190:43121: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
I20260924 19:29:04.566746 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.566658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:38779)
0924 19:29:04.566703 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.566729 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:29:04.566988 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:04.566923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:40141)
0924 19:29:04.566942 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:04.566959 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260924 19:29:04.567083 2144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:29:04.567162 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2015
I20260924 19:29:05.375417 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.375205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:59575)
0924 19:29:05.375341 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:05.375371 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260924 19:29:05.437248 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.437063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:42613)
0924 19:29:05.437175 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:05.437204 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260924 19:29:05.567592 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.567393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:48739)
0924 19:29:05.567507 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:05.567543 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 19:29:05.922086 2151 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:05.922253 2151 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:05.922614 2151 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
Sep 24 19:29:05 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278144, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.131@KRBTEST.COM
Sep 24 19:29:05 dist-test-slave-r8cc krb5kdc[578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278144, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.130@KRBTEST.COM
I20260924 19:29:05.934820 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.922872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:59215 (local address 127.18.229.131:39259)
0924 19:29:05.923046 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:05.923054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:05.923087 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:29:05.923668 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:05.923672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:05.923730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:05.923888 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:05.924852 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.926234 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.927741 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.928383 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.928495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:05.931118 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:05.931123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:05.931123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:05.931134 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:05.933180 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933631 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:05.933634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:05.933635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:05.933727 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933979 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:05.933980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:05.933981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:05.934213 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:05.934304 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:05.934587 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:29:05.934714 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260924 19:29:05.934808 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.922760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:39259 (local address 127.18.229.129:59215)
0924 19:29:05.922918 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:05.922930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:05.922991 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:05.923949 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:05.923952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:05.924455 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:05.924731 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:05.924739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.926352 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.926352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:05.927653 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:05.927656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.928484 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.928485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:05.928626 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:05.929241 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:05.929245 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:05.930995 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:05.933255 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:05.933263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:05.933529 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:05.933794 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:05.933795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:05.933905 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:05.934284 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:05.934290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:05.934415 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 19:29:05.934521 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:29:05.934701 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":54,"thread_start_us":74,"threads_started":1}
I20260924 19:29:05.934842 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.922909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:41225 (local address 127.18.229.130:41319)
0924 19:29:05.923039 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:05.923054 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:05.923088 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:29:05.923668 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:05.923671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:05.923730 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:05.923888 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:05.924851 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.926234 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.927741 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.928383 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.928495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:05.931399 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:05.931403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:05.931403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:05.931414 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:05.933180 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933628 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:05.933630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:05.933631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:05.933725 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933981 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:05.933982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:05.933982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:05.934213 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:05.934304 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:05.934582 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 19:29:05.934727 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":45,"threads_started":1}
I20260924 19:29:05.934913 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:05.922795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41319 (local address 127.18.229.129:41225)
0924 19:29:05.922919 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:05.922930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:05.922991 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:05.923949 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:05.923952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:05.924455 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:05.924731 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:05.924739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.926344 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.926344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:05.927653 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:05.927656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:05.928483 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:05.928484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:05.928626 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:05.929225 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:05.929236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:05.931303 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:05.933306 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:05.933308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:05.933533 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:05.933795 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:05.933796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:05.933797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:05.933911 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:05.934286 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:05.934292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:05.934415 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 19:29:05.934522 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:29:05.934777 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":198,"thread_start_us":58,"threads_started":1}
W20260924 19:29:05.935611 1765 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
W20260924 19:29:05.936184 1766 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
I20260924 19:29:05.936252 1766 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb433dec8341427083b5ea07e90a2137; no voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069
I20260924 19:29:05.936357 2151 raft_consensus.cc:2726] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:29:06.376276 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:06.375962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:37557)
0924 19:29:06.376196 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:06.376235 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260924 19:29:06.437816 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:06.437646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:46335)
0924 19:29:06.437747 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:06.437781 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260924 19:29:06.568310 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:06.568069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:41279)
0924 19:29:06.568242 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:06.568279 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260924 19:29:07.376912 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:07.376721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:37423)
0924 19:29:07.376830 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:07.376866 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260924 19:29:07.438397 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:07.438209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:58801)
0924 19:29:07.438321 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:07.438359 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 19:29:07.477799 2160 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:07.477912 2160 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:07.478106 2160 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
W20260924 19:29:07.478485 1765 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
W20260924 19:29:07.478583 1766 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
I20260924 19:29:07.478680 1765 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb433dec8341427083b5ea07e90a2137; no voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069
I20260924 19:29:07.478787 2160 raft_consensus.cc:2726] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:29:07.568913 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:07.568722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:40033)
0924 19:29:07.568846 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:07.568878 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260924 19:29:08.377565 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:08.377298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:47381)
0924 19:29:08.377441 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:08.377481 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260924 19:29:08.439045 2163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:08.438831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:58485)
0924 19:29:08.438969 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:08.438995 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260924 19:29:08.569540 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:08.569346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:47435)
0924 19:29:08.569459 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:08.569493 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260924 19:29:09.378230 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:09.378035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:56021)
0924 19:29:09.378152 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:09.378186 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 19:29:09.439644 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:09.439443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:36309)
0924 19:29:09.439571 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:09.439603 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260924 19:29:09.570156 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:09.569961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:55249)
0924 19:29:09.570079 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:09.570115 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260924 19:29:10.379050 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:10.378732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:53755)
0924 19:29:10.378949 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:10.379003 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":62,"threads_started":1}
I20260924 19:29:10.440344 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:10.440068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:36699)
0924 19:29:10.440246 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:10.440286 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
I20260924 19:29:10.570866 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:10.570638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:56229)
0924 19:29:10.570778 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:10.570822 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":55,"threads_started":1}
I20260924 19:29:10.665723 2171 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:10.665827 2171 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:10.666049 2171 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
W20260924 19:29:10.666463 1765 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
W20260924 19:29:10.666541 1766 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
I20260924 19:29:10.666600 1765 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb433dec8341427083b5ea07e90a2137; no voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069
I20260924 19:29:10.666673 2171 raft_consensus.cc:2726] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:29:11.379812 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:11.379587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:37497)
0924 19:29:11.379726 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:11.379766 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260924 19:29:11.440964 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:11.440746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:35877)
0924 19:29:11.440894 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:11.440925 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260924 19:29:11.571508 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:11.571308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:47905)
0924 19:29:11.571432 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:11.571468 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 19:29:12.380563 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:12.380328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:39447)
0924 19:29:12.380480 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:12.380518 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260924 19:29:12.441555 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:12.441360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:55791)
0924 19:29:12.441480 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:12.441517 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260924 19:29:12.572278 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:12.572063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:52153)
0924 19:29:12.572211 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:12.572244 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260924 19:29:13.264225 2178 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:13.264344 2178 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:13.264532 2178 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
W20260924 19:29:13.264911 1766 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
W20260924 19:29:13.265017 1765 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
I20260924 19:29:13.265074 1765 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb433dec8341427083b5ea07e90a2137; no voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069
I20260924 19:29:13.265161 2178 raft_consensus.cc:2726] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:29:13.381307 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:13.381062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:37659)
0924 19:29:13.381223 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:13.381262 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260924 19:29:13.442138 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:13.441932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:51607)
0924 19:29:13.442061 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:13.442096 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260924 19:29:13.572914 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:13.572713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:45147)
0924 19:29:13.572825 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:13.572856 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 19:29:14.382017 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:14.381805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:51387)
0924 19:29:14.381932 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:14.381965 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 19:29:14.442868 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:14.442606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:48391)
0924 19:29:14.442770 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:14.442810 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 19:29:14.573556 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:14.573333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:47327)
0924 19:29:14.573452 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:14.573490 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260924 19:29:15.382669 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:15.382473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:43159)
0924 19:29:15.382592 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:15.382625 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260924 19:29:15.443560 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:15.443353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:37077)
0924 19:29:15.443472 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:15.443517 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 19:29:15.574251 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:15.573997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:33941)
0924 19:29:15.574161 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:15.574202 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260924 19:29:16.383462 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:16.383243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:41171)
0924 19:29:16.383381 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:16.383418 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260924 19:29:16.444221 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:16.443968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:41441)
0924 19:29:16.444096 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:16.444175 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260924 19:29:16.575150 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:16.574864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:56297)
0924 19:29:16.575061 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:16.575105 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260924 19:29:17.384272 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:17.384048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:55117)
0924 19:29:17.384193 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:17.384227 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260924 19:29:17.444869 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:17.444678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:47487)
0924 19:29:17.444794 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:17.444825 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260924 19:29:17.575842 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:17.575590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:42943)
0924 19:29:17.575753 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:17.575794 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260924 19:29:18.385087 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:18.384857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:46081)
0924 19:29:18.385002 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:18.385043 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260924 19:29:18.445492 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:18.445288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:44733)
0924 19:29:18.445412 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:18.445449 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260924 19:29:18.557157 2196 raft_consensus.cc:482] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:18.557250 2196 raft_consensus.cc:504] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0e3e934cf440c932f795f629aca30" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 39259 } } peers { permanent_uuid: "dfe591806cb747f08a64a82097b98069" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41319 } } peers { permanent_uuid: "bb433dec8341427083b5ea07e90a2137" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 41263 } }
I20260924 19:29:18.557408 2196 leader_election.cc:288] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259), dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319)
W20260924 19:29:18.557798 1766 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55f0e3e934cf440c932f795f629aca30 (127.18.229.131:39259): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
W20260924 19:29:18.557891 1765 leader_election.cc:341] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfe591806cb747f08a64a82097b98069 (127.18.229.130:41319): Not found: Tablet not found: 48786404ed884cd699f904bcf54bfff5
I20260924 19:29:18.557924 1765 leader_election.cc:302] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb433dec8341427083b5ea07e90a2137; no voters: 55f0e3e934cf440c932f795f629aca30, dfe591806cb747f08a64a82097b98069
I20260924 19:29:18.558001 2196 raft_consensus.cc:2726] T 48786404ed884cd699f904bcf54bfff5 P bb433dec8341427083b5ea07e90a2137 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:29:18.576545 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:18.576342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:54925)
0924 19:29:18.576456 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:18.576502 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260924 19:29:19.385856 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:19.385647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.129:43173)
0924 19:29:19.385773 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:19.385814 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260924 19:29:19.446173 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:19.445962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.130:41121)
0924 19:29:19.446102 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:19.446132 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 19:29:19.569785 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 1719
I20260924 19:29:19.574681 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 1883
I20260924 19:29:19.577235 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:19.577043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:43121 (local address 127.18.229.131:41567)
0924 19:29:19.577162 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:19.577195 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260924 19:29:19.578970 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2015
2026-09-24T19:29:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18188 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:29:19 dist-test-slave-r8cc krb5kdc[2203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:29:19 dist-test-slave-r8cc krb5kdc[2203](info): set up 2 sockets
Sep 24 19:29:19 dist-test-slave-r8cc krb5kdc[2203](info): commencing operation
krb5kdc: starting...
W20260924 19:29:20.628180 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:29:20 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:29:20.746268 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37657
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:20.825891 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:20.826131 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:20.826165 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:20.827311 2216 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:29:20.827376 2216 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:29:20.827395 2216 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:29:20.827414 2216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:29:20.827445 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:20.827466 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:20.827484 2216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:29:20.827503 2216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:29:20.828943 2216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:20.829216 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:20.829423 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:20.834168 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:20.834177 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:20.834265 2216 server_base.cc:1034] running on GCE node
W20260924 19:29:20.834221 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:20.834640 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:20.834954 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:20.836095 2216 hybrid_clock.cc:648] HybridClock initialized: now 1790278160836070 us; error 39 us; skew 500 ppm
Sep 24 19:29:20 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278160, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:20.838646 2216 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:29:20.839061 2216 webserver.cc:460] Webserver started at http://127.18.229.190:44913/ using document root <none> and password file <none>
I20260924 19:29:20.839299 2216 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:20.839367 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:20.839519 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:20.840656 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "a5e96d69301249fa86ded2f575bf6e34"
format_stamp: "Formatted at 2026-09-24 19:29:20 on dist-test-slave-r8cc"
server_key: "f26d48dabbfcd4a6520c5fbe8092b928"
server_key_iv: "628c6036666ce8fd29774cf77977112f"
server_key_version: "encryptionkey@0"
I20260924 19:29:20.841006 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "a5e96d69301249fa86ded2f575bf6e34"
format_stamp: "Formatted at 2026-09-24 19:29:20 on dist-test-slave-r8cc"
server_key: "f26d48dabbfcd4a6520c5fbe8092b928"
server_key_iv: "628c6036666ce8fd29774cf77977112f"
server_key_version: "encryptionkey@0"
I20260924 19:29:20.842278 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:29:20.842983 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:20.843145 2216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:20.843243 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "a5e96d69301249fa86ded2f575bf6e34"
format_stamp: "Formatted at 2026-09-24 19:29:20 on dist-test-slave-r8cc"
server_key: "f26d48dabbfcd4a6520c5fbe8092b928"
server_key_iv: "628c6036666ce8fd29774cf77977112f"
server_key_version: "encryptionkey@0"
I20260924 19:29:20.843315 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:20.862632 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:20.865067 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:20.865235 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:20.869199 2216 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:37657
I20260924 19:29:20.869235 2283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:37657 every 8 connection(s)
I20260924 19:29:20.869613 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:29:20.870508 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:20.870937 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2216
I20260924 19:29:20.871048 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:29:20.871253 19350 external_mini_cluster.cc:1428] Setting key d84762f091d6fe8c78267594aab89302
I20260924 19:29:20.873646 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap starting.
I20260924 19:29:20.874259 2284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:20.874589 2284 log.cc:824] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:20.875319 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: No bootstrap required, opened a new log
I20260924 19:29:20.877872 2284 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:20.878021 2284 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:20.878042 2284 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e96d69301249fa86ded2f575bf6e34, State: Initialized, Role: FOLLOWER
I20260924 19:29:20.878173 2284 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:20.878247 2284 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 24 19:29:20 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:20.878269 2284 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:29:20.878336 2284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:20.878961 2284 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:20.879076 2284 leader_election.cc:302] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e96d69301249fa86ded2f575bf6e34; no voters:
I20260924 19:29:20.879241 2284 leader_election.cc:288] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:29:20.879295 2289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:29:20.879458 2289 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 LEADER]: Becoming Leader. State: Replica: a5e96d69301249fa86ded2f575bf6e34, State: Running, Role: LEADER
I20260924 19:29:20.879516 2284 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:29:20.879590 2289 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:20.880062 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e96d69301249fa86ded2f575bf6e34. Latest consensus state: current_term: 1 leader_uuid: "a5e96d69301249fa86ded2f575bf6e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } } }
I20260924 19:29:20.880038 2290 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5e96d69301249fa86ded2f575bf6e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } } }
I20260924 19:29:20.880226 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:20.880234 2290 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:20.880512 2298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:29:20.881095 2298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:29:20.882773 2298 catalog_manager.cc:1347] Generated new cluster ID: 4db20585432149358faae8b72e2bde98
I20260924 19:29:20.882822 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:20.871814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.18.229.190:37657)
0924 19:29:20.872021 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:20.872031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:20.872065 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:29:20.872527 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:20.872534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:20.872583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:20.872701 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:20.873475 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:20.874450 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:20.875497 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:20.875904 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:20.875997 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:20.878745 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:20.878750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:20.878750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:20.878761 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:20.880976 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:20.881421 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:20.881424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:20.881424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:20.881534 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:20.881793 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:20.881794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:20.881795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:20.882096 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:20.882250 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:20.882495 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 19:29:20.882637 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260924 19:29:20.882822 2298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:29:20.894002 2298 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:29:20.894539 2298 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:29:20.898591 2298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Generated new TSK 0
I20260924 19:29:20.898779 2298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:29:20.928408 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37657
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:21.010154 2310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:21.010349 2310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:21.010371 2310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:21.011507 2310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:21.011545 2310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:21.011570 2310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:29:21.013113 2310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:21.013290 2310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:21.013480 2310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:21.018256 2316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:21.018252 2315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:21.018321 2310 server_base.cc:1034] running on GCE node
W20260924 19:29:21.018247 2318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:21.018669 2310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:21.018877 2310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:21.020037 2310 hybrid_clock.cc:648] HybridClock initialized: now 1790278161020033 us; error 32 us; skew 500 ppm
Sep 24 19:29:21 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278161, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:21.022557 2310 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:29:21.023226 2310 webserver.cc:460] Webserver started at http://127.18.229.129:33931/ using document root <none> and password file <none>
I20260924 19:29:21.023414 2310 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:21.023456 2310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:21.023546 2310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:21.024626 2310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "96da6fc8c8bb370bb0b25a9b5b3c2935"
server_key_iv: "cee7e40c7a1515b29e959f16e888300e"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.024921 2310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "96da6fc8c8bb370bb0b25a9b5b3c2935"
server_key_iv: "cee7e40c7a1515b29e959f16e888300e"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.026154 2310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:29:21.026904 2325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.027076 2310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:21.027160 2310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "96da6fc8c8bb370bb0b25a9b5b3c2935"
server_key_iv: "cee7e40c7a1515b29e959f16e888300e"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.027207 2310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:21.060436 2310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:21.062294 2310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:21.062402 2310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:21.062587 2310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:21.062834 2310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:21.062865 2310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.062886 2310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:21.062899 2310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.068552 2310 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:39343
I20260924 19:29:21.068629 2438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:39343 every 8 connection(s)
I20260924 19:29:21.068949 2310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:29:21.073457 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2310
I20260924 19:29:21.073582 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:29:21.073812 19350 external_mini_cluster.cc:1428] Setting key bcf045e2e2911d219a9870b17116031f
Sep 24 19:29:21 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278161, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:21.078401 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.069740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:46485 (local address 127.18.229.190:37657)
0924 19:29:21.069807 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.069810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.070022 (+ 212us) server_negotiation.cc:407] Connection header received
0924 19:29:21.070486 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.070487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.070496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.070545 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:21.071446 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.071954 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.072959 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.073169 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.073220 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.075536 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:21.075538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:21.075539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:21.075546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:21.077050 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.077453 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.077454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.077455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.077516 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.077764 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.077765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.077766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.077880 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:21.077950 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.078225 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.078313 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:29:21.078603 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.069726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:46485)
0924 19:29:21.069897 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.069935 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.070407 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.070600 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.070603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.071177 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:21.071370 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.071375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.072042 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.072043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.072889 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.072891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.073243 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.073243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.073358 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.073962 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:21.073988 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:21.075438 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:21.077136 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:21.077144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:21.077146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:21.077363 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:21.077582 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:21.077583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:21.077584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:21.077696 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:21.077954 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:21.077961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:21.078098 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.078203 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.078354 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260924 19:29:21.079083 2439 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37657
I20260924 19:29:21.079200 2439 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:21.079408 2439 heartbeater.cc:511] Master 127.18.229.190:37657 requested a full tablet report, sending...
I20260924 19:29:21.079927 2248 ts_manager.cc:194] Registered new tserver with Master: e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:29:21.080788 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:46485
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:29:21.100086 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37657
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:21.181026 2444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:21.181221 2444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:21.181243 2444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:21.182334 2444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:21.182366 2444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:21.182391 2444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:29:21.183790 2444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:21.183979 2444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:21.184172 2444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:21.189109 2449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:21.189159 2444 server_base.cc:1034] running on GCE node
W20260924 19:29:21.189117 2450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:21.189121 2452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:21.189435 2444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:21.189700 2444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:21.190861 2444 hybrid_clock.cc:648] HybridClock initialized: now 1790278161190860 us; error 36 us; skew 500 ppm
Sep 24 19:29:21 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278161, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:21.193243 2444 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:29:21.193702 2444 webserver.cc:460] Webserver started at http://127.18.229.130:34215/ using document root <none> and password file <none>
I20260924 19:29:21.193936 2444 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:21.194005 2444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:21.194152 2444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:21.195189 2444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "0a39010256a74e608bf5885409de7dd5"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "4c5d5a64dd0a7880dc6870b1f92632d9"
server_key_iv: "07431f2258bd654cc2c58e70a1c5e168"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.195544 2444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "0a39010256a74e608bf5885409de7dd5"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "4c5d5a64dd0a7880dc6870b1f92632d9"
server_key_iv: "07431f2258bd654cc2c58e70a1c5e168"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.196910 2444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:29:21.197608 2459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.197779 2444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:21.197894 2444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "0a39010256a74e608bf5885409de7dd5"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "4c5d5a64dd0a7880dc6870b1f92632d9"
server_key_iv: "07431f2258bd654cc2c58e70a1c5e168"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.197966 2444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:21.211716 2444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:21.213492 2444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:21.213647 2444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:21.213860 2444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:21.214147 2444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:21.214201 2444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.214242 2444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:21.214282 2444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.220186 2444 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:40685
I20260924 19:29:21.220256 2572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:40685 every 8 connection(s)
I20260924 19:29:21.220546 2444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:29:21.225128 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2444
I20260924 19:29:21.225227 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:29:21.225457 19350 external_mini_cluster.cc:1428] Setting key 6677704ef72052aaf6425a9bd30c18f3
Sep 24 19:29:21 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278161, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:21.230727 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.221230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:38569 (local address 127.18.229.190:37657)
0924 19:29:21.221286 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.221288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.221494 (+ 206us) server_negotiation.cc:407] Connection header received
0924 19:29:21.222005 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.222006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.222018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.222083 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:21.223000 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.223534 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.224738 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.224995 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.225075 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.227781 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:21.227782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:21.227783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:21.227790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:21.229176 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.229734 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.229735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.229736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.229797 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.230054 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.230055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.230056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.230168 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:21.230246 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.230515 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.230594 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:29:21.230911 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.221208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:38569)
0924 19:29:21.221354 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.221395 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.221895 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.222159 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.222163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.222712 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:21.222896 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.222900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.223634 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.223635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.224630 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.224633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.225098 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.225099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.225242 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.225870 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:21.225880 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:21.227664 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:21.229276 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:21.229283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:21.229285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:21.229596 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:21.229864 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:21.229865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:21.229865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:21.229985 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:21.230237 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:21.230243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:21.230383 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.230493 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.230654 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260924 19:29:21.231622 2573 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37657
I20260924 19:29:21.231729 2573 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:21.231907 2573 heartbeater.cc:511] Master 127.18.229.190:37657 requested a full tablet report, sending...
I20260924 19:29:21.232322 2248 ts_manager.cc:194] Registered new tserver with Master: 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685)
I20260924 19:29:21.232839 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:38569
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:29:21.250480 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37657
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:21.329056 2578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:21.329284 2578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:21.329319 2578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:21.330560 2578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:21.330627 2578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:21.330672 2578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:29:21.332118 2578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:21.332412 2578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:21.332640 2578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:21.337486 2584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:21.337476 2586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:21.337697 2578 server_base.cc:1034] running on GCE node
W20260924 19:29:21.337476 2583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:21.338001 2578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:21.338208 2578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:21.339355 2578 hybrid_clock.cc:648] HybridClock initialized: now 1790278161339333 us; error 26 us; skew 500 ppm
Sep 24 19:29:21 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278161, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:21.341847 2578 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:29:21.342295 2578 webserver.cc:460] Webserver started at http://127.18.229.131:42429/ using document root <none> and password file <none>
I20260924 19:29:21.342504 2578 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:21.342571 2578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:21.342712 2578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:29:21.343731 2578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "8c09c1357ca4e64049d12ea17c46b91e"
server_key_iv: "f5ea806936af26b7968d715aba3dd3c3"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.344100 2578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "8c09c1357ca4e64049d12ea17c46b91e"
server_key_iv: "f5ea806936af26b7968d715aba3dd3c3"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.345436 2578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:29:21.346148 2593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.346323 2578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:29:21.346421 2578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "8c09c1357ca4e64049d12ea17c46b91e"
server_key_iv: "f5ea806936af26b7968d715aba3dd3c3"
server_key_version: "encryptionkey@0"
I20260924 19:29:21.346491 2578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:21.364563 2578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:21.366302 2578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:21.366457 2578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:21.366681 2578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:21.366967 2578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:21.367020 2578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.367053 2578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:21.367080 2578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:21.373047 2578 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:36725
I20260924 19:29:21.373170 2706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:36725 every 8 connection(s)
I20260924 19:29:21.373471 2578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:29:21.374963 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2578
I20260924 19:29:21.375097 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:29:21.375278 19350 external_mini_cluster.cc:1428] Setting key a623eb1f568ecc6a63fb048b566c9334
Sep 24 19:29:21 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278161, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:21.382730 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.374266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:45765 (local address 127.18.229.190:37657)
0924 19:29:21.374331 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.374332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.374498 (+ 166us) server_negotiation.cc:407] Connection header received
0924 19:29:21.374983 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.374984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.374997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.375081 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:21.376387 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.376862 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.377810 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.377999 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.378048 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.380183 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:21.380185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:21.380185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:21.380193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:21.381651 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.382009 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.382010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.382010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.382060 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.382226 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.382226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.382227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.382325 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:21.382375 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.382604 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.382666 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:29:21.383013 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.374235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:45765)
0924 19:29:21.374376 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.374415 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.374878 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.375262 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.375266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.376053 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:21.376309 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.376314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.376938 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.376938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.377738 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.377740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.378075 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.378076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.378168 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.378624 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:21.378631 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:21.380075 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:21.381720 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:21.381726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:21.381728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:21.381942 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:21.382099 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:21.382099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:21.382099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:21.382178 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:21.382380 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:21.382385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:21.382492 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.382578 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.382786 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260924 19:29:21.383551 2707 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37657
I20260924 19:29:21.383649 2707 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:21.383826 2707 heartbeater.cc:511] Master 127.18.229.190:37657 requested a full tablet report, sending...
I20260924 19:29:21.384219 2248 ts_manager.cc:194] Registered new tserver with Master: d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725)
I20260924 19:29:21.384608 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:45765
I20260924 19:29:21.386883 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:29:21.394104 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.388176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.18.229.190:37657)
0924 19:29:21.388236 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.388238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.388302 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:29:21.388374 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.388375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.388384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.388435 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:21.388951 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.389441 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.390182 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.390386 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.390468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.391690 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:21.391691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:21.391692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:21.391698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:21.392920 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.393280 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.393281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.393282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.393335 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:21.393543 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:21.393544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:21.393544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:21.393689 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:21.393788 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.393918 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.393997 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:29:21.395728 2248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:29:21.397356 2248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:29:21.403765 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.399646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:36725 (local address 127.0.0.1:47040)
0924 19:29:21.399872 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.399882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.399956 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.400291 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.400292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.400294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:21.400334 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.400335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.401628 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.401629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.402556 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.402558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.403426 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.403427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.403504 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.403517 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.403626 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.403677 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":46,"threads_started":1}
I20260924 19:29:21.403784 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.399730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.18.229.131:36725)
0924 19:29:21.400100 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.400123 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.400133 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:29:21.400169 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.400171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.400187 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.400404 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:21.400458 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.401524 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.402643 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.403333 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.403411 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.403462 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.403654 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.403699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":74,"threads_started":1}
I20260924 19:29:21.403872 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.399579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.18.229.130:40685)
0924 19:29:21.399796 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.399801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.399811 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:29:21.399850 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.399852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.399868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.400210 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:21.400297 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.401441 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.402860 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.403541 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.403623 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.403672 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.403745 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.403786 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
I20260924 19:29:21.403877 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.399455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:40685 (local address 127.0.0.1:50330)
0924 19:29:21.399576 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.399616 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.399690 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.399971 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.399974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.399977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:21.400054 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.400056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.401566 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.401567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.402773 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.402776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.403617 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.403618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.403674 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.403676 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.403733 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.403766 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":5,"thread_start_us":52,"threads_started":1}
I20260924 19:29:21.404330 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.399783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.0.0.1:33148)
0924 19:29:21.400022 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.400030 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.400066 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.400396 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.400397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.400399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:21.400456 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.400458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.401412 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.401413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.402773 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.402776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.404075 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.404076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.404163 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.404166 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.404227 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.404267 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":29,"thread_start_us":34,"threads_started":1}
I20260924 19:29:21.404662 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.399892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.18.229.129:39343)
0924 19:29:21.400236 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.400241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.400250 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:29:21.400288 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.400290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.400304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.400401 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:21.400557 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.401287 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.402875 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.403345 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.403438 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.403491 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.404521 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.404578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":84,"threads_started":1}
I20260924 19:29:21.404753 2507 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f (DEFAULT_TABLE table=test-table [id=1a8da40a874e4fdcb193186374a9b74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:21.404750 2641 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f (DEFAULT_TABLE table=test-table [id=1a8da40a874e4fdcb193186374a9b74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:21.405085 2641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfbad98b80c4f6ab22c8852661c6d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:21.405086 2507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfbad98b80c4f6ab22c8852661c6d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:21.405176 2373 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f (DEFAULT_TABLE table=test-table [id=1a8da40a874e4fdcb193186374a9b74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:21.405406 2373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfbad98b80c4f6ab22c8852661c6d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:21.407910 2721 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap starting.
I20260924 19:29:21.408408 2723 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Bootstrap starting.
I20260924 19:29:21.408608 2721 tablet_bootstrap.cc:654] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:21.408876 2723 tablet_bootstrap.cc:654] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:21.409164 2721 log.cc:824] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:21.409394 2723 log.cc:824] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:21.409718 2722 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap starting.
I20260924 19:29:21.409917 2721 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: No bootstrap required, opened a new log
I20260924 19:29:21.410000 2721 ts_tablet_manager.cc:1397] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:21.410302 2722 tablet_bootstrap.cc:654] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:21.410741 2722 log.cc:824] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:21.411243 2723 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: No bootstrap required, opened a new log
I20260924 19:29:21.411296 2723 ts_tablet_manager.cc:1397] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:29:21.411537 2722 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: No bootstrap required, opened a new log
I20260924 19:29:21.411635 2722 ts_tablet_manager.cc:1397] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:21.411979 2721 raft_consensus.cc:348] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.412182 2721 raft_consensus.cc:374] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:21.412225 2721 raft_consensus.cc:729] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e932096a3a2e4ed19bafaf44b54d8d5d, State: Initialized, Role: FOLLOWER
I20260924 19:29:21.412333 2721 consensus_queue.cc:260] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.412601 2721 ts_tablet_manager.cc:1428] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:29:21.412735 2439 heartbeater.cc:503] Master 127.18.229.190:37657 was elected leader, sending a full tablet report...
I20260924 19:29:21.413210 2723 raft_consensus.cc:348] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.413357 2723 raft_consensus.cc:374] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:21.413396 2723 raft_consensus.cc:729] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39010256a74e608bf5885409de7dd5, State: Initialized, Role: FOLLOWER
I20260924 19:29:21.413358 2722 raft_consensus.cc:348] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.413504 2722 raft_consensus.cc:374] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:21.413511 2723 consensus_queue.cc:260] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.413558 2722 raft_consensus.cc:729] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f7b9ffe8544df7bf9d1ce2566ba537, State: Initialized, Role: FOLLOWER
I20260924 19:29:21.413676 2722 consensus_queue.cc:260] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.413811 2573 heartbeater.cc:503] Master 127.18.229.190:37657 was elected leader, sending a full tablet report...
I20260924 19:29:21.413932 2707 heartbeater.cc:503] Master 127.18.229.190:37657 was elected leader, sending a full tablet report...
I20260924 19:29:21.413933 2722 ts_tablet_manager.cc:1428] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:21.413802 2723 ts_tablet_manager.cc:1428] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 19:29:21.471110 2574 tablet.cc:2369] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:29:21.569813 2440 tablet.cc:2369] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:29:21.624058 2708 tablet.cc:2369] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:29:21.634601 2729 raft_consensus.cc:482] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:21.634711 2729 raft_consensus.cc:504] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.635025 2729 leader_election.cc:288] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685), e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:29:21.639003 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.635452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:59263 (local address 127.18.229.129:39343)
0924 19:29:21.635502 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.635504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.635612 (+ 108us) server_negotiation.cc:407] Connection header received
0924 19:29:21.635655 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.635656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.635665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.635734 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:21.635862 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.636577 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.638083 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.638649 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.638726 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.638734 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.638894 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.638931 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:29:21.639005 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.635354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:50881 (local address 127.18.229.130:40685)
0924 19:29:21.635420 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.635422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.635432 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:29:21.635460 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.635461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.635474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.635546 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:21.635714 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.636530 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.638084 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.638649 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.638727 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.638734 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.638892 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.638931 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:29:21.639027 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.635215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:40685 (local address 127.18.229.131:50881)
0924 19:29:21.635290 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.635304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.635368 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.635564 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.635565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.635567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:21.635642 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.635644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.636606 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.636607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.637989 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.637992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.638734 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.638735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.638816 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.638817 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.638874 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.638918 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":58}
I20260924 19:29:21.639034 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.635336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.18.229.131:59263)
0924 19:29:21.635532 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.635542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.635599 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.635748 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:21.635748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:21.635750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:21.635803 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.635805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.636646 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.636646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.637989 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:21.637992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.638734 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.638735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:21.638813 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.638817 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:29:21.638874 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:29:21.638917 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":71,"thread_start_us":97,"threads_started":1}
I20260924 19:29:21.639442 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbad98b80c4f6ab22c8852661c6d9f" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" is_pre_election: true
I20260924 19:29:21.639446 2527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbad98b80c4f6ab22c8852661c6d9f" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39010256a74e608bf5885409de7dd5" is_pre_election: true
I20260924 19:29:21.639618 2527 raft_consensus.cc:2454] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 0.
I20260924 19:29:21.639621 2393 raft_consensus.cc:2454] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 0.
I20260924 19:29:21.639839 2597 leader_election.cc:302] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8f7b9ffe8544df7bf9d1ce2566ba537, e932096a3a2e4ed19bafaf44b54d8d5d; no voters:
I20260924 19:29:21.639987 2729 raft_consensus.cc:2781] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:29:21.640045 2729 raft_consensus.cc:482] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:29:21.640069 2729 raft_consensus.cc:3037] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:21.640805 2729 raft_consensus.cc:504] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.640940 2729 leader_election.cc:288] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 election: Requested vote from peers 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685), e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:29:21.641094 2527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbad98b80c4f6ab22c8852661c6d9f" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39010256a74e608bf5885409de7dd5"
I20260924 19:29:21.641121 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbad98b80c4f6ab22c8852661c6d9f" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
I20260924 19:29:21.641177 2527 raft_consensus.cc:3037] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:21.641187 2393 raft_consensus.cc:3037] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:21.642079 2527 raft_consensus.cc:2454] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 1.
I20260924 19:29:21.642079 2393 raft_consensus.cc:2454] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 1.
I20260924 19:29:21.642254 2597 leader_election.cc:302] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a39010256a74e608bf5885409de7dd5, d8f7b9ffe8544df7bf9d1ce2566ba537; no voters:
I20260924 19:29:21.642351 2729 raft_consensus.cc:2781] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:29:21.642520 2729 raft_consensus.cc:686] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 LEADER]: Becoming Leader. State: Replica: d8f7b9ffe8544df7bf9d1ce2566ba537, State: Running, Role: LEADER
I20260924 19:29:21.642627 2729 consensus_queue.cc:237] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:21.643393 2248 catalog_manager.cc:5495] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } health_report { overall_health: UNKNOWN } } }
I20260924 19:29:21.685006 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.682515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.18.229.131:36725)
0924 19:29:21.682574 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:21.682577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:21.682596 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:29:21.682635 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:21.682636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:21.682658 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:21.682736 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:29:21.682867 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.683368 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.684162 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:21.684377 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:21.684463 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:21.684654 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:29:21.684762 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:29:21.684852 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:21.684872 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 19:29:21.684893 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:29:21.686863 2527 raft_consensus.cc:1261] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Refusing update from remote peer d8f7b9ffe8544df7bf9d1ce2566ba537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:21.686863 2393 raft_consensus.cc:1261] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Refusing update from remote peer d8f7b9ffe8544df7bf9d1ce2566ba537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:21.687278 2731 consensus_queue.cc:1035] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:21.687438 2729 consensus_queue.cc:1035] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:21.690213 2735 mvcc.cc:205] Tried to move back new op lower bound from 7332979350266499072 to 7332979350088732672. Current Snapshot: MvccSnapshot[applied={T|T < 7332979350266499072}]
I20260924 19:29:21.692229 2247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:21.692313 2247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:21.693053 2247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Sending DeleteTablet for 3 replicas of tablet 4bfbad98b80c4f6ab22c8852661c6d9f
I20260924 19:29:21.693352 2507 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:21 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50330
I20260924 19:29:21.693382 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2310
I20260924 19:29:21.693455 2641 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:21 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:47040
I20260924 19:29:21.694377 2743 tablet_replica.cc:330] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: stopping tablet replica
I20260924 19:29:21.694475 2743 raft_consensus.cc:2229] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:21.694579 2743 raft_consensus.cc:2258] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:21.695045 2743 ts_tablet_manager.cc:1905] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:21.697067 2743 ts_tablet_manager.cc:1918] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:21.697142 2743 log.cc:1196] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/4bfbad98b80c4f6ab22c8852661c6d9f
I20260924 19:29:21.697402 2743 ts_tablet_manager.cc:1939] T 4bfbad98b80c4f6ab22c8852661c6d9f P 0a39010256a74e608bf5885409de7dd5: Deleting consensus metadata
I20260924 19:29:21.698143 2235 catalog_manager.cc:4841] TS 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685): tablet 4bfbad98b80c4f6ab22c8852661c6d9f (table test-table [id=1a8da40a874e4fdcb193186374a9b74e]) successfully deleted
W20260924 19:29:21.699874 2235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:39343 (error 108)
I20260924 19:29:21.700284 2744 tablet_replica.cc:330] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: stopping tablet replica
I20260924 19:29:21.700423 2744 raft_consensus.cc:2229] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 LEADER]: Raft consensus shutting down.
W20260924 19:29:21.700502 2719 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:29:21.700231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.0.0.1:33156)
0924 19:29:21.700283 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.700294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:21.700332 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:21.700448 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:29:21.700588 2744 raft_consensus.cc:2258] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:29:21.700608 2235 catalog_manager.cc:4584] TS e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bfbad98b80c4f6ab22c8852661c6d9f: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:29:21.700927 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2444
I20260924 19:29:21.707428 2744 ts_tablet_manager.cc:1905] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:21.707506 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2578
I20260924 19:29:21.711987 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2216
I20260924 19:29:21.712146 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:21.711965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:36725 (local address 127.0.0.1:47058)
0924 19:29:21.712017 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:21.712084 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.131:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:29:41.717166 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37657
--webserver_interface=127.18.229.190
--webserver_port=44913
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:29:41.798198 2746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:41.798388 2746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:41.798411 2746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:41.799558 2746 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:29:41.799595 2746 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:29:41.799608 2746 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:29:41.799620 2746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:29:41.799633 2746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:41.799646 2746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:41.799665 2746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:29:41.799677 2746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:29:41.801120 2746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:41.801294 2746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:41.801520 2746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:29:41.809798 2746 server_base.cc:1034] running on GCE node
W20260924 19:29:41.809767 2753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:41.809835 2752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:41.810036 2755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:41.810294 2746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:41.810552 2746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:41.811693 2746 hybrid_clock.cc:648] HybridClock initialized: now 1790278181811681 us; error 27 us; skew 500 ppm
Sep 24 19:29:41 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:41.814163 2746 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:29:41.814586 2746 webserver.cc:460] Webserver started at http://127.18.229.190:44913/ using document root <none> and password file <none>
I20260924 19:29:41.814831 2746 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:41.814909 2746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:41.816280 2746 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:41.816891 2762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:41.817060 2746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:41.817164 2746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "a5e96d69301249fa86ded2f575bf6e34"
format_stamp: "Formatted at 2026-09-24 19:29:20 on dist-test-slave-r8cc"
server_key: "f26d48dabbfcd4a6520c5fbe8092b928"
server_key_iv: "628c6036666ce8fd29774cf77977112f"
server_key_version: "encryptionkey@0"
I20260924 19:29:41.817552 2746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:41.827008 2746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:41.828771 2746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:41.828927 2746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:41.832660 2746 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:37657
I20260924 19:29:41.832692 2814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:37657 every 8 connection(s)
I20260924 19:29:41.833058 2746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:29:41.834453 2815 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:29:41.835084 2815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap starting.
I20260924 19:29:41.842061 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2746
I20260924 19:29:41.842427 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:39343
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:37657
--webserver_port=33931
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:29:41.845979 2815 log.cc:824] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:41.849071 2815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:41.849546 2815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap complete.
I20260924 19:29:41.853652 2815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:41.854122 2815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e96d69301249fa86ded2f575bf6e34, State: Initialized, Role: FOLLOWER
I20260924 19:29:41.854390 2815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:41.854530 2815 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:29:41.854620 2815 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:29:41.854737 2815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:29:41.856139 2815 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:41.856303 2815 leader_election.cc:302] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e96d69301249fa86ded2f575bf6e34; no voters:
I20260924 19:29:41.856503 2815 leader_election.cc:288] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:29:41.856557 2819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:29:41.856741 2819 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 LEADER]: Becoming Leader. State: Replica: a5e96d69301249fa86ded2f575bf6e34, State: Running, Role: LEADER
I20260924 19:29:41.856935 2819 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:29:41.857239 2820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a5e96d69301249fa86ded2f575bf6e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } } }
I20260924 19:29:41.857364 2820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:41.856807 2815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:29:41.857923 2823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:29:41.857239 2819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e96d69301249fa86ded2f575bf6e34. Latest consensus state: current_term: 2 leader_uuid: "a5e96d69301249fa86ded2f575bf6e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } } }
I20260924 19:29:41.858676 2819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:29:41.858769 2823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:29:41.858906 2823 catalog_manager.cc:1259] Loaded cluster ID: 4db20585432149358faae8b72e2bde98
I20260924 19:29:41.859006 2823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:29:41.860256 2823 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:29:41.860460 2823 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Loaded TSK: 0
I20260924 19:29:41.872416 2823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Generated new TSK 1
I20260924 19:29:41.872571 2823 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:29:41.935993 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:41.936285 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:41.936311 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:41.937491 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:41.937531 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:41.937558 2817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:29:41.939044 2817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:39343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=33931
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:41.939275 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:41.939532 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:41.945307 2843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:41.945307 2842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:41.945529 2817 server_base.cc:1034] running on GCE node
W20260924 19:29:41.945291 2845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:41.945858 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:41.946125 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:41.947286 2817 hybrid_clock.cc:648] HybridClock initialized: now 1790278181947275 us; error 30 us; skew 500 ppm
Sep 24 19:29:41 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:41.950074 2817 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:29:41.950547 2817 webserver.cc:460] Webserver started at http://127.18.229.129:33931/ using document root <none> and password file <none>
I20260924 19:29:41.950781 2817 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:41.950865 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:41.952273 2817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:41.952992 2852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:41.953174 2817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:41.953274 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "96da6fc8c8bb370bb0b25a9b5b3c2935"
server_key_iv: "cee7e40c7a1515b29e959f16e888300e"
server_key_version: "encryptionkey@0"
I20260924 19:29:41.953621 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:41.970985 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:41.973220 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:41.973393 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:41.973623 2817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:41.974134 2859 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:29:41.975643 2817 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:29:41.975695 2817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:29:41.975729 2817 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:29:41.976487 2817 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:29:41.976532 2817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:41.976572 2859 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap starting.
I20260924 19:29:41.984233 2817 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:39343
I20260924 19:29:41.984730 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:29:41.988371 2966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:39343 every 8 connection(s)
I20260924 19:29:41.988478 2859 log.cc:824] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:41.988543 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2817
I20260924 19:29:41.988838 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:40685
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:37657
--webserver_port=34215
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:29:41.991003 2859 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:41.991410 2859 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap complete.
I20260924 19:29:41.992655 2859 ts_tablet_manager.cc:1397] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 19:29:41.994123 2859 raft_consensus.cc:348] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:41.994444 2859 raft_consensus.cc:729] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e932096a3a2e4ed19bafaf44b54d8d5d, State: Initialized, Role: FOLLOWER
I20260924 19:29:41.994608 2859 consensus_queue.cc:260] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:41.994890 2859 ts_tablet_manager.cc:1428] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 24 19:29:41 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:42.001224 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:41.985487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:39783)
0924 19:29:41.985688 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:41.985727 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:41.986513 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:41.990640 (+ 4127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:41.990644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:41.990989 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:41.991261 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:41.991267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:41.993096 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:41.993098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:41.994331 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:41.994334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:41.995219 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:41.995219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:41.995323 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:41.995902 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:41.995910 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:41.997313 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:41.999555 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:41.999562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:41.999564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:41.999781 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.000057 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.000057 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.000057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.000203 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.000754 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:42.000760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:42.000867 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.000959 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.001092 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260924 19:29:42.001374 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:41.985573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:39783 (local address 127.18.229.190:37657)
0924 19:29:41.988233 (+ 2660us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:41.988242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:41.988280 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:29:41.990358 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:41.990366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:41.990429 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:41.991388 (+ 959us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:41.991485 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:41.992966 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:41.994449 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:41.995122 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:41.995246 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:41.997429 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:41.997434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:41.997435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:41.997446 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:41.999451 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:41.999863 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:41.999866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:41.999867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:41.999983 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.000267 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.000268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.000268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.000581 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.000679 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.001060 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.001190 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2609,"thread_start_us":76,"threads_started":1}
I20260924 19:29:42.002090 2967 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37657
I20260924 19:29:42.002168 2967 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:42.002350 2967 heartbeater.cc:511] Master 127.18.229.190:37657 requested a full tablet report, sending...
I20260924 19:29:42.003036 2779 ts_manager.cc:194] Registered new tserver with Master: e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:29:42.004197 2779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:39783
I20260924 19:29:42.005097 2967 heartbeater.cc:503] Master 127.18.229.190:37657 was elected leader, sending a full tablet report...
Sep 24 19:29:42 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
I20260924 19:29:42.011528 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.003660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.18.229.129:39343)
0924 19:29:42.003807 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.003811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.003873 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:29:42.003935 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.003937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.003951 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.004022 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:42.004503 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.005414 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.006242 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.006706 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.006786 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.008841 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:42.008847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:42.008847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:42.008852 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:42.010391 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.010800 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.010802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.010802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.010852 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.011045 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.011046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.011046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.011168 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.011225 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.011400 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.011470 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260924 19:29:42.011590 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.003596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.0.0.1:49622)
0924 19:29:42.003775 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.003803 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.003867 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.004036 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.004041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.004373 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:42.004434 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.004436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.005495 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.005496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.006147 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.006151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.006781 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.006782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.006832 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.007286 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:42.007291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:42.008762 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:42.010465 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.010472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.010474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.010723 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.010915 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.010916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.010917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.010986 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.011228 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:42.011234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:42.011283 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.011367 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.011488 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260924 19:29:42.011953 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:21 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:49622
I20260924 19:29:42.012569 2976 tablet_replica.cc:330] stopping tablet replica
I20260924 19:29:42.012669 2976 raft_consensus.cc:2229] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:42.012735 2976 raft_consensus.cc:2258] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:42.013015 2976 ts_tablet_manager.cc:1905] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:42.015102 2976 ts_tablet_manager.cc:1918] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:42.015161 2976 log.cc:1196] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal/wals/4bfbad98b80c4f6ab22c8852661c6d9f
I20260924 19:29:42.015385 2976 ts_tablet_manager.cc:1939] T 4bfbad98b80c4f6ab22c8852661c6d9f P e932096a3a2e4ed19bafaf44b54d8d5d: Deleting consensus metadata
I20260924 19:29:42.015837 2766 catalog_manager.cc:4841] TS e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343): tablet 4bfbad98b80c4f6ab22c8852661c6d9f (table test-table [id=1a8da40a874e4fdcb193186374a9b74e]) successfully deleted
W20260924 19:29:42.075481 2972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:42.075670 2972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:42.075693 2972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:42.076769 2972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:42.076804 2972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:42.076829 2972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:29:42.078183 2972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:40685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=34215
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:42.078356 2972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:42.078516 2972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:42.083372 2982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:42.083339 2981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:42.083356 2972 server_base.cc:1034] running on GCE node
W20260924 19:29:42.083340 2984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:42.083833 2972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:42.084053 2972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:42.085213 2972 hybrid_clock.cc:648] HybridClock initialized: now 1790278182085191 us; error 37 us; skew 500 ppm
Sep 24 19:29:42 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:42.087702 2972 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:29:42.088204 2972 webserver.cc:460] Webserver started at http://127.18.229.130:34215/ using document root <none> and password file <none>
I20260924 19:29:42.088389 2972 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:42.088443 2972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:42.089656 2972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:29:42.090188 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:42.090346 2972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:29:42.090411 2972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "0a39010256a74e608bf5885409de7dd5"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "4c5d5a64dd0a7880dc6870b1f92632d9"
server_key_iv: "07431f2258bd654cc2c58e70a1c5e168"
server_key_version: "encryptionkey@0"
I20260924 19:29:42.090695 2972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:42.103487 2972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:42.105687 2972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:42.105790 2972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:42.105954 2972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:42.106182 2972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:29:42.106210 2972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:42.106230 2972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:29:42.106243 2972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:42.112004 2972 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:40685
I20260924 19:29:42.112076 3104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:40685 every 8 connection(s)
I20260924 19:29:42.112404 2972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:29:42.115223 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 2972
I20260924 19:29:42.115573 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:36725
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:37657
--webserver_port=42429
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:29:42 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:42.122534 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.113101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:34019)
0924 19:29:42.113251 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.113285 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.113737 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.113933 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.113937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.114488 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:42.114665 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.114669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.115325 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.115327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.116660 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.116663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.117136 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.117137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.117237 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.117718 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:42.117726 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:42.119228 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:42.121202 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.121209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.121211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.121481 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.121689 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.121689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.121689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.121781 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.122064 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:42.122068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:42.122188 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.122271 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.122384 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260924 19:29:42.122548 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.113112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:34019 (local address 127.18.229.190:37657)
0924 19:29:42.113166 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.113169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.113370 (+ 201us) server_negotiation.cc:407] Connection header received
0924 19:29:42.113816 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.113817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.113830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.113880 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:42.114736 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.115221 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.116742 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.117044 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.117120 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.119356 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:42.119358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:42.119358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:42.119367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:42.121110 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.121550 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.121552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.121553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.121616 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.121846 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.121847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.121848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.121995 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.122072 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.122317 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.122421 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:29:42.123068 3105 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37657
I20260924 19:29:42.123178 3105 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:42.123351 3105 heartbeater.cc:511] Master 127.18.229.190:37657 requested a full tablet report, sending...
I20260924 19:29:42.123740 2779 ts_manager.cc:194] Registered new tserver with Master: 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685)
I20260924 19:29:42.124285 2779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:34019
W20260924 19:29:42.197644 3108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:29:42.197842 3108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:29:42.197865 3108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:29:42.198995 3108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:29:42.199034 3108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:29:42.199102 3108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:29:42.200515 3108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:36725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=42429
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:29:42.200695 3108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:29:42.200888 3108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:29:42.206120 3116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:42.206171 3114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:29:42.206121 3113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:29:42.206440 3108 server_base.cc:1034] running on GCE node
I20260924 19:29:42.206642 3108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:29:42.206875 3108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:29:42.208041 3108 hybrid_clock.cc:648] HybridClock initialized: now 1790278182208037 us; error 35 us; skew 500 ppm
Sep 24 19:29:42 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:29:42.210639 3108 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:29:42.211053 3108 webserver.cc:460] Webserver started at http://127.18.229.131:42429/ using document root <none> and password file <none>
I20260924 19:29:42.211287 3108 fs_manager.cc:359] Metadata directory not provided
I20260924 19:29:42.211364 3108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:29:42.212740 3108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:29:42.213444 3123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:29:42.213609 3108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:29:42.213706 3108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "8c09c1357ca4e64049d12ea17c46b91e"
server_key_iv: "f5ea806936af26b7968d715aba3dd3c3"
server_key_version: "encryptionkey@0"
I20260924 19:29:42.214020 3108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:29:42.241526 3108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:29:42.244001 3108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:29:42.244194 3108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:29:42.244434 3108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:29:42.244863 3130 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:29:42.246250 3108 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:29:42.246308 3108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:29:42.246354 3108 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:29:42.246977 3108 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:29:42.247030 3108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:42.247080 3130 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap starting.
I20260924 19:29:42.248695 3130 log.cc:824] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:42.252045 3130 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:29:42.252395 3130 tablet_bootstrap.cc:492] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap complete.
I20260924 19:29:42.252556 3130 ts_tablet_manager.cc:1397] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260924 19:29:42.253834 3108 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:36725
I20260924 19:29:42.253865 3238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:36725 every 8 connection(s)
I20260924 19:29:42.254187 3108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:29:42.254640 3130 raft_consensus.cc:348] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.255023 3130 raft_consensus.cc:729] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f7b9ffe8544df7bf9d1ce2566ba537, State: Initialized, Role: FOLLOWER
I20260924 19:29:42.255363 3130 consensus_queue.cc:260] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.255640 3130 ts_tablet_manager.cc:1428] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
Sep 24 19:29:42 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278182, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:29:42.261422 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3108
I20260924 19:29:42.263648 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.254911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:58013 (local address 127.18.229.190:37657)
0924 19:29:42.254984 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.254987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.255194 (+ 207us) server_negotiation.cc:407] Connection header received
0924 19:29:42.255350 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.255351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.255362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.255447 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:42.256213 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.256796 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.257865 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.258053 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.258112 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.260622 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:42.260624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:42.260624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:42.260630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:42.262237 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.262716 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.262717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.262717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.262764 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.263043 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.263044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.263044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.263161 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.263232 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.263476 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.263555 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:29:42.263732 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.254858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:58013)
0924 19:29:42.255073 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.255116 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.255270 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.255529 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.255533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.255901 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:42.256080 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.256084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.256907 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.256907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.257765 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.257767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.258124 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.258125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.258223 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.258740 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:42.258747 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:42.260528 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:42.262311 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.262318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.262321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.262642 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.262829 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.262830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.262831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.262980 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.263232 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:42.263237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:42.263352 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.263436 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.263551 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260924 19:29:42.264295 3239 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37657
I20260924 19:29:42.264400 3239 heartbeater.cc:464] Registering TS with master...
I20260924 19:29:42.264562 3239 heartbeater.cc:511] Master 127.18.229.190:37657 requested a full tablet report, sending...
I20260924 19:29:42.265117 2779 ts_manager.cc:194] Registered new tserver with Master: d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725)
I20260924 19:29:42.265699 2779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:58013
I20260924 19:29:42.266590 3239 heartbeater.cc:503] Master 127.18.229.190:37657 was elected leader, sending a full tablet report...
I20260924 19:29:42.269312 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.261855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.18.229.190:37657)
0924 19:29:42.261977 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.261979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.262064 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:29:42.262094 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.262095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.262107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.262179 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:42.262952 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.263630 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.264705 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.264987 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.265080 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.266643 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:42.266644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:42.266645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:42.266651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:42.268245 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.268629 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.268630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.268631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.268680 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.268858 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.268858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.268858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.268975 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.269100 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.269191 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.269251 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
Sep 24 19:29:42 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278181, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.131@KRBTEST.COM
I20260924 19:29:42.270121 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:29:42.273519 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.265438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:36725 (local address 127.0.0.1:42678)
0924 19:29:42.265615 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.265627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.265670 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.265860 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.265861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.266017 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:29:42.266052 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.266054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.267103 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.267104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.267817 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.267820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.268507 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.268508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.268574 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.268937 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:29:42.268941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:29:42.270503 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:29:42.272432 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.272432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.272433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.272633 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.272831 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:29:42.272832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:29:42.272832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:29:42.272896 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:29:42.273205 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:29:42.273206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:29:42.273250 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.273334 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.273431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260924 19:29:42.273583 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.265511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.18.229.131:36725)
0924 19:29:42.265673 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.265678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.265705 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:29:42.265740 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.265742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.265755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.265853 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:42.266151 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.267004 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.267895 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.268438 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.268486 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.270571 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:42.270579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:42.270579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:42.270586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:42.272350 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.272716 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.272718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.272718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.272768 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.272955 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.272955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.272956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.273126 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.273203 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.273369 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.273457 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
W20260924 19:29:42.273636 2711 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.229.190:37657)
I20260924 19:29:42.273895 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.270471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.18.229.190:37657)
0924 19:29:42.270505 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.270507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.270623 (+ 116us) server_negotiation.cc:407] Connection header received
0924 19:29:42.270690 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.270691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.270715 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.270773 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:29:42.270902 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.271517 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.272617 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.272946 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.273015 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.273219 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:29:42.273264 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 19:29:42.273418 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48924: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:29:42.274152 3244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48924: token expired
I20260924 19:29:42.273942 3173 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbad98b80c4f6ab22c8852661c6d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:21 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:42678
I20260924 19:29:42.274681 3247 tablet_replica.cc:330] stopping tablet replica
I20260924 19:29:42.275393 3247 raft_consensus.cc:2229] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:42.275475 3247 raft_consensus.cc:2258] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:42.275784 3247 ts_tablet_manager.cc:1905] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:42.278345 3247 ts_tablet_manager.cc:1918] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:42.278425 3247 log.cc:1196] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/4bfbad98b80c4f6ab22c8852661c6d9f
I20260924 19:29:42.278672 3247 ts_tablet_manager.cc:1939] T 4bfbad98b80c4f6ab22c8852661c6d9f P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting consensus metadata
I20260924 19:29:42.279155 2766 catalog_manager.cc:4841] TS d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725): tablet 4bfbad98b80c4f6ab22c8852661c6d9f (table test-table [id=1a8da40a874e4fdcb193186374a9b74e]) successfully deleted
I20260924 19:29:42.280555 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.273874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.18.229.190:37657)
0924 19:29:42.273916 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.273918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.274094 (+ 176us) server_negotiation.cc:407] Connection header received
0924 19:29:42.274129 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.274129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.274136 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.274206 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:29:42.274826 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.275266 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.276211 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.276527 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.276591 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.277931 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:29:42.277932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:29:42.277932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:29:42.277939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:29:42.279497 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.279806 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.279808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.279808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.279861 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:29:42.280066 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:29:42.280067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:29:42.280067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:29:42.280199 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:29:42.280301 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.280397 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.280466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:29:42.281549 2779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:29:42.283001 2779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:29:42.284929 3173 tablet_service.cc:1467] Processing CreateTablet for tablet be0c918ac0674464890d18b70e098506 (DEFAULT_TABLE table=test-table [id=24da7e2c969b48bf8bb89ba34aaad677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:42.284932 2901 tablet_service.cc:1467] Processing CreateTablet for tablet be0c918ac0674464890d18b70e098506 (DEFAULT_TABLE table=test-table [id=24da7e2c969b48bf8bb89ba34aaad677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:42.285082 3173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0c918ac0674464890d18b70e098506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:42.285111 2901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0c918ac0674464890d18b70e098506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:42.286309 3130 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap starting.
I20260924 19:29:42.286466 2859 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap starting.
I20260924 19:29:42.286782 2859 tablet_bootstrap.cc:654] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:42.286791 3130 tablet_bootstrap.cc:654] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:42.287425 2859 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: No bootstrap required, opened a new log
I20260924 19:29:42.287503 3130 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: No bootstrap required, opened a new log
I20260924 19:29:42.287503 2859 ts_tablet_manager.cc:1397] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:42.287583 3130 ts_tablet_manager.cc:1397] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:42.287834 3130 raft_consensus.cc:348] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.287882 2859 raft_consensus.cc:348] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.287957 2859 raft_consensus.cc:374] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:42.288008 2859 raft_consensus.cc:729] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e932096a3a2e4ed19bafaf44b54d8d5d, State: Initialized, Role: FOLLOWER
I20260924 19:29:42.287995 3130 raft_consensus.cc:374] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:42.288059 2859 consensus_queue.cc:260] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.288041 3130 raft_consensus.cc:729] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f7b9ffe8544df7bf9d1ce2566ba537, State: Initialized, Role: FOLLOWER
I20260924 19:29:42.288199 2859 ts_tablet_manager.cc:1428] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:29:42.288210 3130 consensus_queue.cc:260] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.288307 3130 ts_tablet_manager.cc:1428] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:29:42.289883 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.284606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:40685 (local address 127.0.0.1:49550)
0924 19:29:42.284654 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.284665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.284698 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.286613 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.286614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.286616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:42.286654 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.286655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.287912 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.287913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.289013 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.289016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.289684 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.289685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.289732 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.289746 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.289803 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.289833 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:29:42.289897 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.284712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.18.229.130:40685)
0924 19:29:42.286417 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.286421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.286432 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:29:42.286475 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.286477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.286493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.286608 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:42.286705 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.287814 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.289087 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.289613 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.289673 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.289708 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.289813 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.289838 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":100,"threads_started":1}
I20260924 19:29:42.290545 3039 tablet_service.cc:1467] Processing CreateTablet for tablet be0c918ac0674464890d18b70e098506 (DEFAULT_TABLE table=test-table [id=24da7e2c969b48bf8bb89ba34aaad677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:29:42.290851 3039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0c918ac0674464890d18b70e098506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:29:42.293759 3251 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Bootstrap starting.
I20260924 19:29:42.294456 3251 tablet_bootstrap.cc:654] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Neither blocks nor log segments found. Creating new log.
I20260924 19:29:42.294934 3251 log.cc:824] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:29:42.295601 3251 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: No bootstrap required, opened a new log
I20260924 19:29:42.295650 3251 ts_tablet_manager.cc:1397] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:29:42.297060 3251 raft_consensus.cc:348] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.297192 3251 raft_consensus.cc:374] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:29:42.297215 3251 raft_consensus.cc:729] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39010256a74e608bf5885409de7dd5, State: Initialized, Role: FOLLOWER
I20260924 19:29:42.297380 3251 consensus_queue.cc:260] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.297678 3251 ts_tablet_manager.cc:1428] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:29:42.297691 3105 heartbeater.cc:503] Master 127.18.229.190:37657 was elected leader, sending a full tablet report...
W20260924 19:29:42.362946 3106 tablet.cc:2369] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:29:42.423235 3242 raft_consensus.cc:482] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:29:42.423418 3242 raft_consensus.cc:504] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.423738 3242 leader_election.cc:288] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685), e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:29:42.426622 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.424034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:34995 (local address 127.18.229.130:40685)
0924 19:29:42.424097 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.424100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.424131 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:29:42.424219 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.424220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.424233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.424310 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:42.424476 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.424934 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.425865 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.426263 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.426347 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.426356 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.426522 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.426559 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:29:42.426640 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.423966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:40685 (local address 127.18.229.131:34995)
0924 19:29:42.424022 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.424034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.424091 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.424317 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.424318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.424320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:42.424401 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.424403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.425014 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.425015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.425771 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.425773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.426345 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.426346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.426426 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.426444 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.426515 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.426554 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:29:42.426921 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.424190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:55277 (local address 127.18.229.129:39343)
0924 19:29:42.424256 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.424258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.424364 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:29:42.424440 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.424441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.424471 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.424543 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:29:42.424717 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.425186 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.426118 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.426514 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.426609 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.426687 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.426817 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.426857 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:29:42.426921 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.424087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.18.229.131:55277)
0924 19:29:42.424301 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.424308 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:29:42.424376 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:29:42.424550 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:29:42.424550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:29:42.424552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:29:42.424643 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.424645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.425280 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.425281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.426025 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:29:42.426027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.426625 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.426626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:29:42.426687 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.426692 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:29:42.426778 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:29:42.426824 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260924 19:29:42.426951 3059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0c918ac0674464890d18b70e098506" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39010256a74e608bf5885409de7dd5" is_pre_election: true
I20260924 19:29:42.427115 3059 raft_consensus.cc:2454] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 0.
I20260924 19:29:42.427309 3127 leader_election.cc:302] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a39010256a74e608bf5885409de7dd5, d8f7b9ffe8544df7bf9d1ce2566ba537; no voters:
I20260924 19:29:42.427273 2921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0c918ac0674464890d18b70e098506" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" is_pre_election: true
I20260924 19:29:42.427414 2921 raft_consensus.cc:2454] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 0.
I20260924 19:29:42.427498 3242 raft_consensus.cc:2781] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:29:42.427584 3242 raft_consensus.cc:482] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:29:42.427609 3242 raft_consensus.cc:3037] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:42.428373 3242 raft_consensus.cc:504] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.428500 3242 leader_election.cc:288] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 election: Requested vote from peers 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685), e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:29:42.428684 3059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0c918ac0674464890d18b70e098506" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39010256a74e608bf5885409de7dd5"
I20260924 19:29:42.428704 2921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0c918ac0674464890d18b70e098506" candidate_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
I20260924 19:29:42.428766 3059 raft_consensus.cc:3037] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:42.428799 2921 raft_consensus.cc:3037] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:29:42.429469 2921 raft_consensus.cc:2454] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 1.
I20260924 19:29:42.429474 3059 raft_consensus.cc:2454] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f7b9ffe8544df7bf9d1ce2566ba537 in term 1.
I20260924 19:29:42.429626 3127 leader_election.cc:302] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a39010256a74e608bf5885409de7dd5, d8f7b9ffe8544df7bf9d1ce2566ba537; no voters:
I20260924 19:29:42.429728 3242 raft_consensus.cc:2781] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:29:42.429884 3242 raft_consensus.cc:686] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 LEADER]: Becoming Leader. State: Replica: d8f7b9ffe8544df7bf9d1ce2566ba537, State: Running, Role: LEADER
I20260924 19:29:42.429936 3242 consensus_queue.cc:237] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:29:42.430680 2779 catalog_manager.cc:5495] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } health_report { overall_health: UNKNOWN } } }
I20260924 19:29:42.470942 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.468483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.18.229.131:36725)
0924 19:29:42.468545 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:29:42.468547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:29:42.468557 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:29:42.468644 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:29:42.468645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:29:42.468670 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:29:42.468751 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:29:42.468875 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.469375 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.470113 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:29:42.470346 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:29:42.470434 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:29:42.470621 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:29:42.470718 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:29:42.470799 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:29:42.470839 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 19:29:42.470858 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:29:42.472565 3059 raft_consensus.cc:1261] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Refusing update from remote peer d8f7b9ffe8544df7bf9d1ce2566ba537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:42.472898 2921 raft_consensus.cc:1261] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Refusing update from remote peer d8f7b9ffe8544df7bf9d1ce2566ba537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:29:42.472900 3242 consensus_queue.cc:1035] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:42.473138 3242 consensus_queue.cc:1035] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:29:42.476133 3258 mvcc.cc:205] Tried to move back new op lower bound from 7332979435405090816 to 7332979435233832960. Current Snapshot: MvccSnapshot[applied={T|T < 7332979435405090816 or (T in {7332979435405090816})}]
I20260924 19:29:42.477246 2779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:42.477344 2779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:29:42.478078 2779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Sending DeleteTablet for 3 replicas of tablet be0c918ac0674464890d18b70e098506
I20260924 19:29:42.478415 3173 tablet_service.cc:1514] Processing DeleteTablet for tablet be0c918ac0674464890d18b70e098506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:42 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:42678
I20260924 19:29:42.478430 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet be0c918ac0674464890d18b70e098506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:42 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:49622
I20260924 19:29:42.478483 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2817
I20260924 19:29:42.478503 3247 tablet_replica.cc:330] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: stopping tablet replica
I20260924 19:29:42.478545 3247 raft_consensus.cc:2229] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:29:42.478615 3247 raft_consensus.cc:2258] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:42.478684 2976 tablet_replica.cc:330] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: stopping tablet replica
I20260924 19:29:42.478734 2976 raft_consensus.cc:2229] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:42.478776 2976 raft_consensus.cc:2258] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:42.479177 3039 tablet_service.cc:1514] Processing DeleteTablet for tablet be0c918ac0674464890d18b70e098506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:29:42 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:49550
I20260924 19:29:42.479568 3247 ts_tablet_manager.cc:1905] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:29:42.480981 3266 tablet_replica.cc:330] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: stopping tablet replica
I20260924 19:29:42.481119 3266 raft_consensus.cc:2229] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:29:42.481211 3247 ts_tablet_manager.cc:1918] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:29:42.481261 3247 log.cc:1196] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/be0c918ac0674464890d18b70e098506
I20260924 19:29:42.481262 3266 raft_consensus.cc:2258] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:29:42.481498 3247 ts_tablet_manager.cc:1939] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Deleting consensus metadata
I20260924 19:29:42.481789 2766 catalog_manager.cc:4841] TS d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725): tablet be0c918ac0674464890d18b70e098506 (table test-table [id=24da7e2c969b48bf8bb89ba34aaad677]) successfully deleted
I20260924 19:29:42.485526 3266 ts_tablet_manager.cc:1905] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:29:42.485551 2766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:39343 (error 108)
I20260924 19:29:42.485890 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2972
I20260924 19:29:42.491446 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 3108
I20260924 19:29:42.491611 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.491452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.0.0.1:49626)
0924 19:29:42.491524 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.491572 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 19:29:42.491711 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:29:42.491527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:40685 (local address 127.0.0.1:49562)
0924 19:29:42.491667 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:29:42.491685 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":48,"threads_started":1}
W20260924 19:29:42.492420 2766 catalog_manager.cc:4584] TS e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be0c918ac0674464890d18b70e098506: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39343: connect: Connection refused (error 111)
I20260924 19:29:42.495568 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 2746
I20260924 19:30:02.500900 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37657
--webserver_interface=127.18.229.190
--webserver_port=44913
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:02.582470 3269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:02.582662 3269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:02.582684 3269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:02.583772 3269 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:30:02.583804 3269 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:30:02.583817 3269 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:30:02.583828 3269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:02.583841 3269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:02.583856 3269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:02.583887 3269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:02.583899 3269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:02.585325 3269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:02.585572 3269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:02.585775 3269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:30:02.590767 3269 server_base.cc:1034] running on GCE node
W20260924 19:30:02.590775 3274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:02.590751 3277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:02.590751 3275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:02.591262 3269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:02.591552 3269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:02.592741 3269 hybrid_clock.cc:648] HybridClock initialized: now 1790278202592723 us; error 48 us; skew 500 ppm
Sep 24 19:30:02 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:02.595140 3269 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:02.595559 3269 webserver.cc:460] Webserver started at http://127.18.229.190:44913/ using document root <none> and password file <none>
I20260924 19:30:02.595779 3269 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:02.595847 3269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:02.597239 3269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:02.597886 3284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:02.598061 3269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:02.598161 3269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "a5e96d69301249fa86ded2f575bf6e34"
format_stamp: "Formatted at 2026-09-24 19:29:20 on dist-test-slave-r8cc"
server_key: "f26d48dabbfcd4a6520c5fbe8092b928"
server_key_iv: "628c6036666ce8fd29774cf77977112f"
server_key_version: "encryptionkey@0"
I20260924 19:30:02.598524 3269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:02.609491 3269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:02.611179 3269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:02.611339 3269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:02.615083 3269 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:37657
I20260924 19:30:02.615118 3336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:37657 every 8 connection(s)
I20260924 19:30:02.615513 3269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:02.616040 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3269
I20260924 19:30:02.616351 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:39343
--local_ip_for_outbound_sockets=127.18.229.129
--tserver_master_addrs=127.18.229.190:37657
--webserver_port=33931
--webserver_interface=127.18.229.129
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:02.617177 3337 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:30:02.617830 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap starting.
I20260924 19:30:02.626158 3337 log.cc:824] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:02.628469 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:30:02.628816 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Bootstrap complete.
I20260924 19:30:02.631000 3337 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:30:02.631259 3337 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e96d69301249fa86ded2f575bf6e34, State: Initialized, Role: FOLLOWER
I20260924 19:30:02.631471 3337 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:30:02.631577 3337 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:02.631631 3337 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:02.631680 3337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:30:02.632778 3337 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:30:02.632902 3337 leader_election.cc:302] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e96d69301249fa86ded2f575bf6e34; no voters:
I20260924 19:30:02.633116 3337 leader_election.cc:288] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:30:02.633150 3341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:30:02.633268 3341 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [term 3 LEADER]: Becoming Leader. State: Replica: a5e96d69301249fa86ded2f575bf6e34, State: Running, Role: LEADER
I20260924 19:30:02.633394 3337 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:02.633425 3341 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } }
I20260924 19:30:02.633885 3342 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a5e96d69301249fa86ded2f575bf6e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } } }
I20260924 19:30:02.633896 3343 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e96d69301249fa86ded2f575bf6e34. Latest consensus state: current_term: 3 leader_uuid: "a5e96d69301249fa86ded2f575bf6e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96d69301249fa86ded2f575bf6e34" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37657 } } }
I20260924 19:30:02.633996 3342 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:02.634016 3343 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:02.634413 3350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:02.635282 3350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:02.635450 3350 catalog_manager.cc:1259] Loaded cluster ID: 4db20585432149358faae8b72e2bde98
I20260924 19:30:02.635670 3350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:02.636557 3350 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:02.636727 3350 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Loaded TSK: 0
I20260924 19:30:02.636802 3350 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5e96d69301249fa86ded2f575bf6e34: Loaded TSK: 1
F20260924 19:30:02.637056 3350 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe1def3f9ad google::LogMessage::Fail() at ??:0
@ 0x7fe1def4288a google::LogMessage::SendToLog() at ??:0
@ 0x7fe1def3f4de google::LogMessage::Flush() at ??:0
@ 0x7fe1def40a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe1e16ac104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe1e16a9d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe1e6e991cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe1e6e9948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe1e6e99594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe1e6e5fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe1e6e69ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe1e0e0d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe1e0e02954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe1e1c666db start_thread at ??:0
@ 0x7fe1dda3d61f clone at ??:0
W20260924 19:30:02.727347 3339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:02.727610 3339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:02.727645 3339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:02.729288 3339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:02.729346 3339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:02.729384 3339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:02.731387 3339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:39343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=33931
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:02.731630 3339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:02.731911 3339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:02.737278 3381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:02.737291 3379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:02.737401 3378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:02.737599 3339 server_base.cc:1034] running on GCE node
I20260924 19:30:02.737797 3339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:02.738060 3339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:02.739264 3339 hybrid_clock.cc:648] HybridClock initialized: now 1790278202739245 us; error 26 us; skew 500 ppm
Sep 24 19:30:02 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:02.742182 3339 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:30:02.742620 3339 webserver.cc:460] Webserver started at http://127.18.229.129:33931/ using document root <none> and password file <none>
I20260924 19:30:02.742867 3339 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:02.742938 3339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:02.744349 3339 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:02.745085 3388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:02.745256 3339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:02.745369 3339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "96da6fc8c8bb370bb0b25a9b5b3c2935"
server_key_iv: "cee7e40c7a1515b29e959f16e888300e"
server_key_version: "encryptionkey@0"
I20260924 19:30:02.745712 3339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:02.760030 3339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:02.762480 3339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:02.762645 3339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:02.762884 3339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:02.763345 3395 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:30:02.765394 3339 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:30:02.765455 3339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:30:02.765496 3339 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:30:02.766222 3339 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:30:02.766275 3339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:30:02.766356 3395 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap starting.
I20260924 19:30:02.773516 3339 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:39343
I20260924 19:30:02.773592 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:39343 every 8 connection(s)
I20260924 19:30:02.773957 3339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:30:02.777328 3395 log.cc:824] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:02.780176 3395 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:30:02.780512 3395 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Bootstrap complete.
I20260924 19:30:02.781726 3395 ts_tablet_manager.cc:1397] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260924 19:30:02.781821 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3339
I20260924 19:30:02.782115 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:40685
--local_ip_for_outbound_sockets=127.18.229.130
--tserver_master_addrs=127.18.229.190:37657
--webserver_port=34215
--webserver_interface=127.18.229.130
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:02.783219 3395 raft_consensus.cc:348] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:02.783552 3395 raft_consensus.cc:729] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e932096a3a2e4ed19bafaf44b54d8d5d, State: Initialized, Role: FOLLOWER
I20260924 19:30:02.783732 3395 consensus_queue.cc:260] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:02.784029 3395 ts_tablet_manager.cc:1428] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260924 19:30:02.871398 3505 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:02.774691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:38489)
0924 19:30:02.774882 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:02.774928 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:02.775379 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:02.871178 (+ 95799us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
W20260924 19:30:02.871658 3503 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:37657 (0 consecutive failures): Network error: Failed to ping master at 127.18.229.190:37657: Client connection negotiation failed: client connection to 127.18.229.190:37657: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:30:02.872018 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:02.871892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:44469)
0924 19:30:02.871950 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:02.871989 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:30:02.872413 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:02.872326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:44653)
0924 19:30:02.872374 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:02.872397 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 19:30:02.872547 3503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:30:02.892400 3507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:02.892622 3507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:02.892661 3507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:02.893826 3507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:02.893899 3507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:02.893944 3507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:02.895378 3507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:40685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=34215
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:02.895632 3507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:02.895854 3507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:02.900869 3516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:02.900914 3514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:02.900849 3513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:02.901149 3507 server_base.cc:1034] running on GCE node
I20260924 19:30:02.901357 3507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:02.901613 3507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:02.902809 3507 hybrid_clock.cc:648] HybridClock initialized: now 1790278202902794 us; error 41 us; skew 500 ppm
Sep 24 19:30:02 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:02.905507 3507 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:30:02.905885 3507 webserver.cc:460] Webserver started at http://127.18.229.130:34215/ using document root <none> and password file <none>
I20260924 19:30:02.906065 3507 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:02.906106 3507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:02.907331 3507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:02.908021 3523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:02.908202 3507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:02.908282 3507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "0a39010256a74e608bf5885409de7dd5"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "4c5d5a64dd0a7880dc6870b1f92632d9"
server_key_iv: "07431f2258bd654cc2c58e70a1c5e168"
server_key_version: "encryptionkey@0"
I20260924 19:30:02.908593 3507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:02.925333 3507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:02.927117 3507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:02.927220 3507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:02.927387 3507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:02.927728 3530 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:30:02.929194 3507 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:30:02.929235 3507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:30:02.929260 3507 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:30:02.929873 3507 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:30:02.929904 3507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:30:02.930030 3530 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Bootstrap starting.
I20260924 19:30:02.931726 3530 log.cc:824] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:02.934875 3530 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:30:02.935194 3530 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Bootstrap complete.
I20260924 19:30:02.935340 3530 ts_tablet_manager.cc:1397] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 19:30:02.936430 3507 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:40685
I20260924 19:30:02.936493 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:40685 every 8 connection(s)
I20260924 19:30:02.936774 3507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:30:02.937719 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:02.937394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:36755)
0924 19:30:02.937541 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:02.937606 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 19:30:02.937557 3530 raft_consensus.cc:348] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:02.937821 3530 raft_consensus.cc:729] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39010256a74e608bf5885409de7dd5, State: Initialized, Role: FOLLOWER
I20260924 19:30:02.937961 3530 consensus_queue.cc:260] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
W20260924 19:30:02.937965 3639 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:37657 (0 consecutive failures): Network error: Failed to ping master at 127.18.229.190:37657: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
I20260924 19:30:02.938109 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3507
I20260924 19:30:02.938230 3530 ts_tablet_manager.cc:1428] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
I20260924 19:30:02.938302 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:02.938212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:51883)
0924 19:30:02.938250 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:02.938281 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 19:30:02.938416 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:36725
--local_ip_for_outbound_sockets=127.18.229.131
--tserver_master_addrs=127.18.229.190:37657
--webserver_port=42429
--webserver_interface=127.18.229.131
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:02.938580 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:02.938510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:53579)
0924 19:30:02.938541 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:02.938559 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 19:30:02.938679 3639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:30:03.017923 3643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:03.018123 3643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:03.018146 3643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:03.019219 3643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:03.019253 3643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:03.019279 3643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:03.020723 3643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:36725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=42429
--tserver_master_addrs=127.18.229.190:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:03.020902 3643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:03.021068 3643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:03.025907 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:03.026029 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:03.026057 3643 server_base.cc:1034] running on GCE node
W20260924 19:30:03.026161 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:03.026350 3643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:03.026634 3643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:03.027792 3643 hybrid_clock.cc:648] HybridClock initialized: now 1790278203027782 us; error 28 us; skew 500 ppm
Sep 24 19:30:03 dist-test-slave-r8cc krb5kdc[2203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278203, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:03.030153 3643 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:30:03.030568 3643 webserver.cc:460] Webserver started at http://127.18.229.131:42429/ using document root <none> and password file <none>
I20260924 19:30:03.030789 3643 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:03.030856 3643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:03.032215 3643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:03.032855 3658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:03.033025 3643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:03.033118 3643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537"
format_stamp: "Formatted at 2026-09-24 19:29:21 on dist-test-slave-r8cc"
server_key: "8c09c1357ca4e64049d12ea17c46b91e"
server_key_iv: "f5ea806936af26b7968d715aba3dd3c3"
server_key_version: "encryptionkey@0"
I20260924 19:30:03.033476 3643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:03.046007 3643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:03.047756 3643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:03.047914 3643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:03.048210 3643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:03.048604 3643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:03.048655 3643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:03.048697 3643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:03.048723 3643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:03.054530 3643 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:36725
I20260924 19:30:03.054579 3771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:36725 every 8 connection(s)
I20260924 19:30:03.054894 3643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:30:03.055851 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:03.055495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:52531)
0924 19:30:03.055634 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:03.055705 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260924 19:30:03.056308 3772 heartbeater.cc:644] Failed to heartbeat to 127.18.229.190:37657 (0 consecutive failures): Network error: Failed to ping master at 127.18.229.190:37657: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
I20260924 19:30:03.056593 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:03.056506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:32869)
0924 19:30:03.056550 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:03.056576 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:30:03.056898 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:03.056824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:45389)
0924 19:30:03.056862 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:03.056883 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:30:03.057120 3772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:30:03.063236 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3643
I20260924 19:30:03.873037 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:03.872828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:53517)
0924 19:30:03.872949 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:03.872980 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 19:30:03.939160 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:03.938983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:46539)
0924 19:30:03.939086 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:03.939119 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260924 19:30:04.057763 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.057570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:38411)
0924 19:30:04.057683 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:04.057716 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 19:30:04.493944 3779 raft_consensus.cc:482] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:04.494064 3779 raft_consensus.cc:504] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:04.494362 3779 leader_election.cc:288] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725), e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
Sep 24 19:30:04 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.129@KRBTEST.COM
Sep 24 19:30:04 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.131@KRBTEST.COM
I20260924 19:30:04.506165 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.494724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39343 (local address 127.18.229.130:46115)
0924 19:30:04.494851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:04.494859 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:04.494905 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:04.495240 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:04.495246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:04.495527 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:04.495852 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:04.495859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.497477 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.497478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:04.498672 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:04.498674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.499521 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.499522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:04.499674 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:04.500347 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:04.500375 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:04.502406 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:04.504435 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:04.504441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:04.504443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:04.504733 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:04.504977 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:04.504977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:04.504978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:04.505108 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:04.505585 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:04.505592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:04.505746 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 19:30:04.505861 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:30:04.506028 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260924 19:30:04.506184 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.494846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:46115 (local address 127.18.229.129:39343)
0924 19:30:04.494971 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:04.494974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:04.494984 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:04.495054 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:04.495056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:04.495079 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:04.495186 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:04.495985 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.497371 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.498775 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.499403 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.499513 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:04.502544 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:04.502550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:04.502550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:04.502563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:04.504357 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:04.504820 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:04.504822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:04.504822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:04.504911 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:04.505175 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:04.505176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:04.505176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:04.505493 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:04.505581 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:04.505934 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 19:30:04.506080 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":46,"threads_started":1}
I20260924 19:30:04.506494 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.494655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:49983 (local address 127.18.229.131:36725)
0924 19:30:04.494840 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:04.494851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:04.494896 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:30:04.495522 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:04.495525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:04.495564 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:04.495665 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:04.496522 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.497784 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.498772 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.499398 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.499513 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:04.502910 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:04.502914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:04.502914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:04.502926 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:04.504925 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:04.505306 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:04.505308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:04.505308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:04.505386 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:04.505584 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:04.505585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:04.505586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:04.505876 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:04.505950 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:04.506152 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:30:04.506386 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260924 19:30:04.506351 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.494544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:36725 (local address 127.18.229.130:49983)
0924 19:30:04.494679 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:04.494711 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:04.494891 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:04.495678 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:04.495679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:04.496353 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:04.496408 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:04.496411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.497870 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.497871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:04.498666 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:04.498669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:04.499519 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:04.499521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:04.499674 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:04.500411 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:04.500416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:04.502788 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:04.505028 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:04.505029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:04.505029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:04.505230 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:04.505454 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:04.505454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:04.505455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:04.505521 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:04.505949 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:04.505949 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:04.506006 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 19:30:04.506082 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:30:04.506173 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260924 19:30:04.507215 3457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0c918ac0674464890d18b70e098506" candidate_uuid: "0a39010256a74e608bf5885409de7dd5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" is_pre_election: true
I20260924 19:30:04.507371 3457 raft_consensus.cc:2454] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a39010256a74e608bf5885409de7dd5 in term 1.
W20260924 19:30:04.507576 3527 leader_election.cc:341] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725): Not found: Tablet not found: be0c918ac0674464890d18b70e098506
I20260924 19:30:04.507684 3527 leader_election.cc:302] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a39010256a74e608bf5885409de7dd5, e932096a3a2e4ed19bafaf44b54d8d5d; no voters: d8f7b9ffe8544df7bf9d1ce2566ba537
I20260924 19:30:04.507804 3779 raft_consensus.cc:2781] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:30:04.507876 3779 raft_consensus.cc:482] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:04.507894 3779 raft_consensus.cc:3037] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:30:04.509124 3779 raft_consensus.cc:504] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:04.509264 3779 leader_election.cc:288] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [CANDIDATE]: Term 2 election: Requested vote from peers d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725), e932096a3a2e4ed19bafaf44b54d8d5d (127.18.229.129:39343)
I20260924 19:30:04.509490 3457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0c918ac0674464890d18b70e098506" candidate_uuid: "0a39010256a74e608bf5885409de7dd5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d"
W20260924 19:30:04.509552 3527 leader_election.cc:341] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725): Not found: Tablet not found: be0c918ac0674464890d18b70e098506
I20260924 19:30:04.509579 3457 raft_consensus.cc:3037] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:30:04.510643 3457 raft_consensus.cc:2454] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a39010256a74e608bf5885409de7dd5 in term 2.
I20260924 19:30:04.510845 3527 leader_election.cc:302] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a39010256a74e608bf5885409de7dd5, e932096a3a2e4ed19bafaf44b54d8d5d; no voters: d8f7b9ffe8544df7bf9d1ce2566ba537
I20260924 19:30:04.510936 3779 raft_consensus.cc:2781] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:30:04.511053 3779 raft_consensus.cc:686] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [term 2 LEADER]: Becoming Leader. State: Replica: 0a39010256a74e608bf5885409de7dd5, State: Running, Role: LEADER
I20260924 19:30:04.511144 3779 consensus_queue.cc:237] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:04.511409 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.511303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:50377)
0924 19:30:04.511342 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:04.511376 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 19:30:04.873643 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:04.873445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:49075)
0924 19:30:04.873561 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:04.873599 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
W20260924 19:30:04.981202 3527 consensus_peers.cc:487] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 -> Peer d8f7b9ffe8544df7bf9d1ce2566ba537 (127.18.229.131:36725): Couldn't send request to peer d8f7b9ffe8544df7bf9d1ce2566ba537. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be0c918ac0674464890d18b70e098506. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:30:05.058385 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.058187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:33487)
0924 19:30:05.058304 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.058343 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 19:30:05.112556 3457 raft_consensus.cc:1261] T be0c918ac0674464890d18b70e098506 P e932096a3a2e4ed19bafaf44b54d8d5d [term 2 FOLLOWER]: Refusing update from remote peer 0a39010256a74e608bf5885409de7dd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 19:30:05.112926 3779 consensus_queue.cc:1035] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 19:30:05.113780 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.113662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:47535)
0924 19:30:05.113709 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.113751 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:30:05.114744 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.114661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:33849)
0924 19:30:05.114692 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.114724 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 19:30:05.114749 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.114590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:48427)
0924 19:30:05.114663 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.114695 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58,"thread_start_us":39,"threads_started":1}
I20260924 19:30:05.361371 3794 ts_tablet_manager.cc:927] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Initiating tablet copy from peer 0a39010256a74e608bf5885409de7dd5 (127.18.229.130:40685)
I20260924 19:30:05.361590 3794 tablet_copy_client.cc:316] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: tablet copy: Beginning tablet copy session from remote peer at address 127.18.229.130:40685
Sep 24 19:30:05 dist-test-slave-r8cc krb5kdc[2203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278203, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.130@KRBTEST.COM
I20260924 19:30:05.368973 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.362040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:49223 (local address 127.18.229.130:40685)
0924 19:30:05.362223 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:05.362228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:05.362264 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:30:05.362292 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:05.362294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:05.362309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:05.362405 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:05.362843 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:05.363505 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:05.364274 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:05.364568 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:05.364623 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:05.366583 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:05.366590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:05.366590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:05.366596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:05.367925 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:05.368334 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:05.368337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:05.368337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:05.368391 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:05.368546 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:05.368546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:05.368547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:05.368666 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:05.368721 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:05.368848 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:30:05.368905 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260924 19:30:05.368984 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.361896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:40685 (local address 127.18.229.131:49223)
0924 19:30:05.361957 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.361993 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:05.362065 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:05.362426 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:05.362431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:05.362704 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:05.362771 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:05.362773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:05.363572 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:05.363572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:05.364210 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:05.364213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:05.364632 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:05.364632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:05.364684 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:05.365143 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:05.365148 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:05.366494 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:05.367996 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:05.368002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:05.368003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:05.368259 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:05.368450 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:05.368451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:05.368451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:05.368505 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:05.368718 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:05.368723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:05.368767 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:30:05.368829 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:30:05.368912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:30:05.369446 3613 tablet_copy_service.cc:140] P 0a39010256a74e608bf5885409de7dd5: Received BeginTabletCopySession request for tablet be0c918ac0674464890d18b70e098506 from peer d8f7b9ffe8544df7bf9d1ce2566ba537 ({username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:49223)
I20260924 19:30:05.369505 3613 tablet_copy_service.cc:161] P 0a39010256a74e608bf5885409de7dd5: Beginning new tablet copy session on tablet be0c918ac0674464890d18b70e098506 from peer d8f7b9ffe8544df7bf9d1ce2566ba537 at {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:49223: session id = d8f7b9ffe8544df7bf9d1ce2566ba537-be0c918ac0674464890d18b70e098506
I20260924 19:30:05.369989 3613 tablet_copy_source_session.cc:205] T be0c918ac0674464890d18b70e098506 P 0a39010256a74e608bf5885409de7dd5: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:30:05.370899 3794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0c918ac0674464890d18b70e098506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:05.373828 3794 tablet_copy_client.cc:791] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: tablet copy: Starting download of 0 data blocks...
I20260924 19:30:05.373986 3794 tablet_copy_client.cc:655] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: tablet copy: Starting download of 1 WAL segments...
I20260924 19:30:05.374815 3794 tablet_copy_client.cc:531] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:30:05.375455 3794 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap starting.
I20260924 19:30:05.376801 3794 log.cc:824] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:05.379060 3794 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:30:05.379287 3794 tablet_bootstrap.cc:492] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Bootstrap complete.
I20260924 19:30:05.379432 3794 ts_tablet_manager.cc:1397] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:30:05.380443 3794 raft_consensus.cc:348] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:05.380615 3794 raft_consensus.cc:729] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f7b9ffe8544df7bf9d1ce2566ba537, State: Initialized, Role: FOLLOWER
I20260924 19:30:05.380769 3794 consensus_queue.cc:260] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39010256a74e608bf5885409de7dd5" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 40685 } } peers { permanent_uuid: "d8f7b9ffe8544df7bf9d1ce2566ba537" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36725 } } peers { permanent_uuid: "e932096a3a2e4ed19bafaf44b54d8d5d" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39343 } }
I20260924 19:30:05.381026 3794 ts_tablet_manager.cc:1428] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260924 19:30:05.381445 3613 tablet_copy_service.cc:342] P 0a39010256a74e608bf5885409de7dd5: Request end of tablet copy session d8f7b9ffe8544df7bf9d1ce2566ba537-be0c918ac0674464890d18b70e098506 received from {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:49223
I20260924 19:30:05.381485 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.381360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:54265)
0924 19:30:05.381418 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.381454 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:30:05.381510 3613 tablet_copy_service.cc:433] P 0a39010256a74e608bf5885409de7dd5: ending tablet copy session d8f7b9ffe8544df7bf9d1ce2566ba537-be0c918ac0674464890d18b70e098506 on tablet be0c918ac0674464890d18b70e098506 with peer d8f7b9ffe8544df7bf9d1ce2566ba537
I20260924 19:30:05.381807 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.381701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:48451)
0924 19:30:05.381757 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.381786 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 19:30:05.745528 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.745331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:34119)
0924 19:30:05.745431 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.745471 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
I20260924 19:30:05.745512 3726 raft_consensus.cc:1203] T be0c918ac0674464890d18b70e098506 P d8f7b9ffe8544df7bf9d1ce2566ba537 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 19:30:05.746030 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:05.745895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:41625)
0924 19:30:05.745951 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:05.745990 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:30:06.115350 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:06.115149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:48381)
0924 19:30:06.115265 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:06.115303 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260924 19:30:06.746176 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:06.745961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:35681)
0924 19:30:06.746093 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:06.746132 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260924 19:30:06.746562 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:06.746387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:46599)
0924 19:30:06.746488 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:06.746525 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260924 19:30:07.116206 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:07.115948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:52671)
0924 19:30:07.116075 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:07.116160 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 19:30:07.746862 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:07.746624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:43479)
0924 19:30:07.746764 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:07.746802 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 19:30:07.747171 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:07.746976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:50383)
0924 19:30:07.747098 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:07.747135 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260924 19:30:08.116876 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:08.116655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:33841)
0924 19:30:08.116799 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:08.116832 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260924 19:30:08.747718 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:08.747423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:40353)
0924 19:30:08.747585 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:08.747624 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260924 19:30:08.747875 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:08.747633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:44325)
0924 19:30:08.747751 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:08.747797 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260924 19:30:08.748175 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:08.748040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:58971)
0924 19:30:08.748085 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:08.748144 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:30:09.117654 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:09.117419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:44547)
0924 19:30:09.117557 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:09.117606 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260924 19:30:09.748607 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:09.748403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:34787)
0924 19:30:09.748531 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:09.748563 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 19:30:09.748775 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:09.748546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:32923)
0924 19:30:09.748676 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:09.748728 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260924 19:30:10.118330 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:10.118131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:44697)
0924 19:30:10.118246 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:10.118282 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 19:30:10.749326 3815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:10.749028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:34157)
0924 19:30:10.749175 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:10.749265 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260924 19:30:10.749435 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:10.749225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:41497)
0924 19:30:10.749338 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:10.749378 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260924 19:30:11.119150 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:11.118893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:41785)
0924 19:30:11.119057 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:11.119104 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260924 19:30:11.750001 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:11.749759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:58057)
0924 19:30:11.749883 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:11.749939 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260924 19:30:11.750054 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:11.749802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:50439)
0924 19:30:11.749974 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:11.750014 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260924 19:30:12.119937 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:12.119730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:60663)
0924 19:30:12.119857 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:12.119892 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260924 19:30:12.750695 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:12.750490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:39677)
0924 19:30:12.750614 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:12.750652 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260924 19:30:12.750712 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:12.750490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:52317)
0924 19:30:12.750639 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:12.750672 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260924 19:30:13.120584 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:13.120371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:56107)
0924 19:30:13.120504 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:13.120540 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 19:30:13.751346 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:13.751136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:33563)
0924 19:30:13.751274 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:13.751302 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260924 19:30:13.751368 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:13.751168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:59421)
0924 19:30:13.751298 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:13.751329 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260924 19:30:14.121212 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:14.121002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:59977)
0924 19:30:14.121121 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:14.121159 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260924 19:30:14.752058 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:14.751842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:34805)
0924 19:30:14.751971 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:14.752012 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260924 19:30:14.752068 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:14.751876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:56859)
0924 19:30:14.751998 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:14.752029 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260924 19:30:15.121956 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:15.121626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:38933)
0924 19:30:15.121869 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:15.121909 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":58,"threads_started":1}
I20260924 19:30:15.752717 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:15.752520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:59709)
0924 19:30:15.752644 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:15.752673 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 19:30:15.752780 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:15.752577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:54979)
0924 19:30:15.752694 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:15.752740 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260924 19:30:16.122673 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:16.122447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:59603)
0924 19:30:16.122577 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:16.122612 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260924 19:30:16.753407 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:16.753185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:60285)
0924 19:30:16.753324 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:16.753362 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260924 19:30:16.753429 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:16.753209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:41835)
0924 19:30:16.753352 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:16.753391 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260924 19:30:17.123574 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:17.123325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.129:53005)
0924 19:30:17.123454 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:17.123499 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 19:30:17.754070 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:17.753867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.130:38813)
0924 19:30:17.753990 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:17.754025 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260924 19:30:17.754086 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:17.753867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37657 (local address 127.18.229.131:34381)
0924 19:30:17.754018 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:17.754046 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.190:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260924 19:30:18.067636 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 3339
I20260924 19:30:18.072463 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 3507
I20260924 19:30:18.076933 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 3643
2026-09-24T19:30:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58498 ms)
[----------] 30 tests from SecurityITest (230477 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:18 dist-test-slave-r8cc krb5kdc[3840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:18 dist-test-slave-r8cc krb5kdc[3840](info): set up 2 sockets
Sep 24 19:30:18 dist-test-slave-r8cc krb5kdc[3840](info): commencing operation
krb5kdc: starting...
W20260924 19:30:19.111819 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:19.230337 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34623
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:19.310460 3853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:19.310650 3853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:19.310671 3853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:19.311805 3853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:19.311838 3853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:19.311852 3853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:19.311864 3853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:19.311877 3853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:19.313315 3853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:19.313503 3853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:19.313663 3853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:30:19.318336 3853 server_base.cc:1034] running on GCE node
W20260924 19:30:19.318341 3858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.318305 3861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.318284 3859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:19.318753 3853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:19.319042 3853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:19.320179 3853 hybrid_clock.cc:648] HybridClock initialized: now 1790278219320167 us; error 40 us; skew 500 ppm
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:19.322630 3853 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:19.323235 3853 webserver.cc:460] Webserver started at http://127.18.229.190:46699/ using document root <none> and password file <none>
I20260924 19:30:19.323473 3853 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:19.323541 3853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:19.323691 3853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:19.324807 3853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "25cf72c2d0a04af1930344753d1d7984"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "775991cc92f9835a387ec854ed460f2d"
server_key_iv: "3afc6cccd840e8fc1f99d883868f7fdc"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.325166 3853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "25cf72c2d0a04af1930344753d1d7984"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "775991cc92f9835a387ec854ed460f2d"
server_key_iv: "3afc6cccd840e8fc1f99d883868f7fdc"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.326471 3853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:19.327179 3868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.327359 3853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:19.327466 3853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "25cf72c2d0a04af1930344753d1d7984"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "775991cc92f9835a387ec854ed460f2d"
server_key_iv: "3afc6cccd840e8fc1f99d883868f7fdc"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.327538 3853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:19.341853 3853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:19.344216 3853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:19.344380 3853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:19.348224 3853 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:34623
I20260924 19:30:19.348274 3920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:34623 every 8 connection(s)
I20260924 19:30:19.348634 3853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:19.349493 3921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:19.351724 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984: Bootstrap starting.
I20260924 19:30:19.352349 3921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:19.352640 3921 log.cc:824] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:19.353405 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984: No bootstrap required, opened a new log
I20260924 19:30:19.355105 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3853
I20260924 19:30:19.355021 3921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf72c2d0a04af1930344753d1d7984" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34623 } }
I20260924 19:30:19.355189 3921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:19.355211 3921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cf72c2d0a04af1930344753d1d7984, State: Initialized, Role: FOLLOWER
I20260924 19:30:19.355250 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:19.355378 3921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf72c2d0a04af1930344753d1d7984" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34623 } }
I20260924 19:30:19.355446 3921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:19.355466 3921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:19.355508 19350 external_mini_cluster.cc:1428] Setting key 5d73bbe6b8d3a9701254e27ec76c2507
I20260924 19:30:19.355527 3921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:19.356194 3921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf72c2d0a04af1930344753d1d7984" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34623 } }
I20260924 19:30:19.356292 3921 leader_election.cc:302] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25cf72c2d0a04af1930344753d1d7984; no voters:
I20260924 19:30:19.356503 3921 leader_election.cc:288] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:19.356532 3926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:19.356632 3926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [term 1 LEADER]: Becoming Leader. State: Replica: 25cf72c2d0a04af1930344753d1d7984, State: Running, Role: LEADER
I20260924 19:30:19.356755 3921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:19.356724 3926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf72c2d0a04af1930344753d1d7984" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34623 } }
I20260924 19:30:19.357637 3928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25cf72c2d0a04af1930344753d1d7984. Latest consensus state: current_term: 1 leader_uuid: "25cf72c2d0a04af1930344753d1d7984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf72c2d0a04af1930344753d1d7984" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34623 } } }
I20260924 19:30:19.357752 3928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:19.357986 3927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25cf72c2d0a04af1930344753d1d7984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf72c2d0a04af1930344753d1d7984" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34623 } } }
I20260924 19:30:19.358114 3927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:19.358192 3938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:19.358769 3938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:19.360365 3938 catalog_manager.cc:1347] Generated new cluster ID: 116c4fe38ace4d3e91455127bceec1a8
I20260924 19:30:19.360445 3938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:19.366935 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.356318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.18.229.190:34623)
0924 19:30:19.356495 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.356503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.356526 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:30:19.356651 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.356658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.356737 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.356935 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:19.357782 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.359111 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.360169 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.360771 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.360907 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.363210 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:19.363214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:19.363215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:19.363227 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:19.365369 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.365757 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.365760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.365761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.365863 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.366077 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.366078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.366078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.366317 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:19.366483 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.366577 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.366727 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260924 19:30:19.396142 3938 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:19.396874 3938 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:19.409802 3938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984: Generated new TSK 0
I20260924 19:30:19.410027 3938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:30:19.443077 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34623
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:19.524704 3947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:19.524915 3947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:19.524942 3947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:19.526127 3947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:19.526170 3947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:19.526196 3947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:19.527719 3947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:19.527976 3947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:19.528236 3947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:19.533375 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.533375 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.533401 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:19.533579 3947 server_base.cc:1034] running on GCE node
I20260924 19:30:19.533849 3947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:19.534137 3947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:19.535310 3947 hybrid_clock.cc:648] HybridClock initialized: now 1790278219535284 us; error 45 us; skew 500 ppm
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:19.537935 3947 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:30:19.538332 3947 webserver.cc:460] Webserver started at http://127.18.229.129:37857/ using document root <none> and password file <none>
I20260924 19:30:19.538566 3947 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:19.538614 3947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:19.538743 3947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:19.539819 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "98d13ab8499746f3aca4b0fc7655c1ba"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "3ac81e1fc52f6f9dc520ed354291d5b3"
server_key_iv: "6353fc10275c78d8b825e841c094296a"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.540237 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "98d13ab8499746f3aca4b0fc7655c1ba"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "3ac81e1fc52f6f9dc520ed354291d5b3"
server_key_iv: "6353fc10275c78d8b825e841c094296a"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.541590 3947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:19.542333 3962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.542510 3947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:19.542598 3947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "98d13ab8499746f3aca4b0fc7655c1ba"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "3ac81e1fc52f6f9dc520ed354291d5b3"
server_key_iv: "6353fc10275c78d8b825e841c094296a"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.542673 3947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:19.558365 3947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:19.560791 3947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:19.560953 3947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:19.561197 3947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:19.561537 3947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:19.561592 3947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.561627 3947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:19.561654 3947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.567466 3947 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:45983
I20260924 19:30:19.567518 4075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:45983 every 8 connection(s)
I20260924 19:30:19.567826 3947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:19.577261 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.568540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:45957 (local address 127.18.229.190:34623)
0924 19:30:19.568608 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.568610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.568823 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:30:19.569366 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.569367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.569380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.569430 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:19.570364 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.570893 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.571912 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.572196 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.572288 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.574460 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:19.574462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:19.574462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:19.574470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:19.575789 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.576260 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.576262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.576263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.576330 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.576591 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.576592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.576592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.576714 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:19.576791 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.577080 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.577169 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:30:19.577461 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.568549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34623 (local address 127.18.229.129:45957)
0924 19:30:19.568693 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.568730 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.569242 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.569504 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.569507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.570060 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:19.570256 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.570260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.570996 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.570997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.571814 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.571816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.572304 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.572305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.572425 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.572977 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:19.572985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:19.574361 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:19.575881 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:19.575888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:19.575891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:19.576166 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:19.576405 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:19.576406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:19.576406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:19.576518 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:19.576780 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:19.576787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:19.576937 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:30:19.577047 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:30:19.577211 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260924 19:30:19.578065 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 3947
I20260924 19:30:19.578063 4076 heartbeater.cc:347] Connected to a master server at 127.18.229.190:34623
I20260924 19:30:19.578171 4076 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:19.578181 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:30:19.578395 4076 heartbeater.cc:511] Master 127.18.229.190:34623 requested a full tablet report, sending...
I20260924 19:30:19.578416 19350 external_mini_cluster.cc:1428] Setting key 10e23435ef0545b7ef0ac71f68bbff99
I20260924 19:30:19.579069 3884 ts_manager.cc:194] Registered new tserver with Master: 98d13ab8499746f3aca4b0fc7655c1ba (127.18.229.129:45983)
I20260924 19:30:19.579679 3884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:45957
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:30:19.603485 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34623
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:19.682518 4081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:19.682715 4081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:19.682739 4081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:19.683913 4081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:19.683954 4081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:19.683981 4081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:19.685479 4081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:19.685727 4081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:19.685990 4081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:19.691059 4087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.690999 4089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.691051 4086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:19.691020 4081 server_base.cc:1034] running on GCE node
I20260924 19:30:19.691468 4081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:19.691687 4081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:19.692833 4081 hybrid_clock.cc:648] HybridClock initialized: now 1790278219692820 us; error 35 us; skew 500 ppm
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:19.695300 4081 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:30:19.695678 4081 webserver.cc:460] Webserver started at http://127.18.229.130:42849/ using document root <none> and password file <none>
I20260924 19:30:19.695870 4081 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:19.695910 4081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:19.695997 4081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:19.697070 4081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "f1ded23534574f17bf9b676937c541cc"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "de83f6b2ac036401bc4c733597a03c82"
server_key_iv: "cb5c577ce3aa8630cbfeb260b7e85d87"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.697377 4081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "f1ded23534574f17bf9b676937c541cc"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "de83f6b2ac036401bc4c733597a03c82"
server_key_iv: "cb5c577ce3aa8630cbfeb260b7e85d87"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.698770 4081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:19.699607 4096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.699801 4081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:19.699908 4081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "f1ded23534574f17bf9b676937c541cc"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "de83f6b2ac036401bc4c733597a03c82"
server_key_iv: "cb5c577ce3aa8630cbfeb260b7e85d87"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.699982 4081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:19.721963 4081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:19.723965 4081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:19.724192 4081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:19.724439 4081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:19.724767 4081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:19.724822 4081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.724866 4081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:19.724895 4081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.730844 4081 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:33809
I20260924 19:30:19.730877 4209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:33809 every 8 connection(s)
I20260924 19:30:19.731223 4081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:19.738365 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4081
I20260924 19:30:19.738484 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:30:19.738796 19350 external_mini_cluster.cc:1428] Setting key f4a9dc9886294e2b9666591fbd8a16a8
I20260924 19:30:19.740037 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.731952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:53093 (local address 127.18.229.190:34623)
0924 19:30:19.731996 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.731998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.732231 (+ 233us) server_negotiation.cc:407] Connection header received
0924 19:30:19.732711 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.732712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.732721 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.732776 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:19.733683 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.734165 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.735167 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.735344 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.735393 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.737340 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:19.737342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:19.737342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:19.737348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:19.738529 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.739043 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.739045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.739046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.739109 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.739373 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.739374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.739375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.739497 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:19.739564 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.739911 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.739977 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:19.740363 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.731921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34623 (local address 127.18.229.130:53093)
0924 19:30:19.732083 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.732145 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.732627 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.732829 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.732832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.733415 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:19.733606 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.733610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.734245 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.734246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.735094 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.735096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.735414 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.735415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.735526 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.735939 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:19.735946 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:19.737262 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:19.738612 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:19.738619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:19.738622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:19.738964 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:19.739178 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:19.739179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:19.739179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:19.739306 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:19.739561 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:19.739583 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:19.739771 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 19:30:19.739889 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 19:30:19.740064 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260924 19:30:19.740775 4210 heartbeater.cc:347] Connected to a master server at 127.18.229.190:34623
I20260924 19:30:19.740896 4210 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:19.741088 4210 heartbeater.cc:511] Master 127.18.229.190:34623 requested a full tablet report, sending...
I20260924 19:30:19.741521 3884 ts_manager.cc:194] Registered new tserver with Master: f1ded23534574f17bf9b676937c541cc (127.18.229.130:33809)
I20260924 19:30:19.742053 3884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:53093
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:30:19.764461 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34623
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:19.848412 4215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:19.848592 4215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:19.848615 4215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:19.849733 4215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:19.849769 4215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:19.849799 4215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:19.851241 4215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:34623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:19.851419 4215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:19.851583 4215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:19.856441 4223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.856441 4221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:19.856465 4220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:19.856511 4215 server_base.cc:1034] running on GCE node
I20260924 19:30:19.856794 4215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:19.857064 4215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:19.858233 4215 hybrid_clock.cc:648] HybridClock initialized: now 1790278219858220 us; error 26 us; skew 500 ppm
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:19.860703 4215 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:30:19.861126 4215 webserver.cc:460] Webserver started at http://127.18.229.131:37015/ using document root <none> and password file <none>
I20260924 19:30:19.861316 4215 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:19.861356 4215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:19.861436 4215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:19.862365 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "d3a94bf9515cbd8c5ee4cf0834eb043e"
server_key_iv: "a0cfc62669e3eacbee632d5cdf8a9ff9"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.862636 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "d3a94bf9515cbd8c5ee4cf0834eb043e"
server_key_iv: "a0cfc62669e3eacbee632d5cdf8a9ff9"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.863857 4215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:30:19.864584 4230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.864771 4215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.864859 4215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5"
format_stamp: "Formatted at 2026-09-24 19:30:19 on dist-test-slave-r8cc"
server_key: "d3a94bf9515cbd8c5ee4cf0834eb043e"
server_key_iv: "a0cfc62669e3eacbee632d5cdf8a9ff9"
server_key_version: "encryptionkey@0"
I20260924 19:30:19.864934 4215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:19.873934 4215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:19.875602 4215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:19.875749 4215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:19.875959 4215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:19.876322 4215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:19.876355 4215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.876376 4215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:19.876420 4215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:19.881763 4215 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:36839
I20260924 19:30:19.881861 4343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:36839 every 8 connection(s)
I20260924 19:30:19.882141 4215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:30:19 dist-test-slave-r8cc krb5kdc[3840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278219, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:19.889084 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4215
I20260924 19:30:19.889170 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:30:19.889374 19350 external_mini_cluster.cc:1428] Setting key f98361d37b7697a674cee5221ec12e14
I20260924 19:30:19.890753 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.882858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:59831 (local address 127.18.229.190:34623)
0924 19:30:19.882913 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.882915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.883058 (+ 143us) server_negotiation.cc:407] Connection header received
0924 19:30:19.883523 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.883524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.883534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.883590 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:19.884562 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.885051 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.885973 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.886149 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.886197 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.888154 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:19.888156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:19.888156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:19.888170 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:19.889422 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.889816 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.889818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.889818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.889879 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.890128 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.890129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.890130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.890244 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:19.890320 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.890590 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.890665 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:30:19.890934 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.882780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34623 (local address 127.18.229.131:59831)
0924 19:30:19.882934 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.882976 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.883442 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.883662 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.883666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.884291 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:19.884478 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.884484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.885128 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.885128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.885903 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.885905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.886218 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.886219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.886322 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.886758 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:19.886766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:19.888038 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:19.889500 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:19.889507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:19.889509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:19.889730 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:19.889947 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:19.889948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:19.889948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:19.890052 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:19.890313 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:19.890319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:19.890453 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:30:19.890557 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 19:30:19.890710 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 19:30:19.891378 4344 heartbeater.cc:347] Connected to a master server at 127.18.229.190:34623
I20260924 19:30:19.891458 4344 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:19.891661 4344 heartbeater.cc:511] Master 127.18.229.190:34623 requested a full tablet report, sending...
I20260924 19:30:19.892035 3884 ts_manager.cc:194] Registered new tserver with Master: c555fbef7f4b4d3b89a611bd03e2c5b5 (127.18.229.131:36839)
I20260924 19:30:19.892503 3884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:59831
I20260924 19:30:19.901033 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:30:19.908293 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.902342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.18.229.190:34623)
0924 19:30:19.902408 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.902410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.902526 (+ 116us) server_negotiation.cc:407] Connection header received
0924 19:30:19.902594 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.902595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.902606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.902677 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:19.903168 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.903635 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.904573 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.904784 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.904869 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.905970 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:19.905971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:19.905972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:19.905979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:19.907085 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.907448 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.907449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.907450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.907509 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:19.907705 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:19.907706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:19.907706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:19.907860 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:19.907988 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.908095 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.908193 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:30:19.910072 3884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:30:19.911777 3884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:30:19.918449 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.914124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45983 (local address 127.0.0.1:33742)
0924 19:30:19.914557 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.914567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.914610 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.914825 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.914826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.914828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:19.914870 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.914871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.916283 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.916284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.917249 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.917252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.918152 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.918153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.918210 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.918222 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:30:19.918315 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 19:30:19.918370 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":87,"threads_started":1}
I20260924 19:30:19.918483 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.914309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.18.229.129:45983)
0924 19:30:19.914616 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.914619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.914630 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:30:19.914685 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.914687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.914704 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.915027 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:19.915117 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.916189 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.917342 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.918070 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.918151 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.918208 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.918353 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.918398 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260924 19:30:19.918642 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.914385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.18.229.130:33809)
0924 19:30:19.914551 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.914555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.914819 (+ 264us) server_negotiation.cc:407] Connection header received
0924 19:30:19.914915 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.914917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.914932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.914997 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:19.915292 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.915950 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.917529 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.918204 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.918277 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.918346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.918498 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.918543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260924 19:30:19.918787 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.914125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:33809 (local address 127.0.0.1:45350)
0924 19:30:19.914685 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.914693 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.914735 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.915073 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.915074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.915076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:19.915117 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.915119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.916050 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.916051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.917444 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.917446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.918279 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.918280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.918348 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.918351 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:19.918398 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:30:19.918432 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":74,"thread_start_us":77,"threads_started":1}
I20260924 19:30:19.919502 4355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.914114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:36839 (local address 127.0.0.1:33636)
0924 19:30:19.914279 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.914317 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.914381 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.914716 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.914719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.914722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:19.914806 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.914809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.916143 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.916144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.917443 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.917446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.919263 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.919264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:19.919332 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.919337 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:30:19.919401 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:30:19.919440 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":39,"thread_start_us":60,"threads_started":1}
I20260924 19:30:19.919539 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.914388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.18.229.131:36839)
0924 19:30:19.914534 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.914539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.914550 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:30:19.914593 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.914596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.914613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.914697 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:19.914895 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.916033 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.918385 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:19.919115 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:19.919243 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:19.919307 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:19.919414 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 19:30:19.919453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260924 19:30:19.919605 4144 tablet_service.cc:1467] Processing CreateTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e (DEFAULT_TABLE table=test-table [id=6b92814558ad4764820073b738c664be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:19.919601 4010 tablet_service.cc:1467] Processing CreateTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e (DEFAULT_TABLE table=test-table [id=6b92814558ad4764820073b738c664be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:19.920001 4010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75139d76c13b4760970bb5c4c08b4a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:19.920006 4144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75139d76c13b4760970bb5c4c08b4a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:19.920288 4278 tablet_service.cc:1467] Processing CreateTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e (DEFAULT_TABLE table=test-table [id=6b92814558ad4764820073b738c664be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:19.920583 4278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75139d76c13b4760970bb5c4c08b4a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:19.922640 4361 tablet_bootstrap.cc:492] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba: Bootstrap starting.
I20260924 19:30:19.923354 4361 tablet_bootstrap.cc:654] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:19.923379 4362 tablet_bootstrap.cc:492] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Bootstrap starting.
I20260924 19:30:19.923605 4363 tablet_bootstrap.cc:492] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Bootstrap starting.
I20260924 19:30:19.924043 4361 log.cc:824] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:19.924089 4362 tablet_bootstrap.cc:654] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:19.924371 4363 tablet_bootstrap.cc:654] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:19.924628 4362 log.cc:824] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:19.924862 4361 tablet_bootstrap.cc:492] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba: No bootstrap required, opened a new log
I20260924 19:30:19.924928 4361 ts_tablet_manager.cc:1397] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:30:19.925006 4363 log.cc:824] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:19.925367 4362 tablet_bootstrap.cc:492] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: No bootstrap required, opened a new log
I20260924 19:30:19.925438 4362 ts_tablet_manager.cc:1397] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:19.925776 4363 tablet_bootstrap.cc:492] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: No bootstrap required, opened a new log
I20260924 19:30:19.925884 4363 ts_tablet_manager.cc:1397] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:19.926968 4361 raft_consensus.cc:348] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.927145 4361 raft_consensus.cc:374] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:19.927182 4361 raft_consensus.cc:729] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d13ab8499746f3aca4b0fc7655c1ba, State: Initialized, Role: FOLLOWER
I20260924 19:30:19.927243 4363 raft_consensus.cc:348] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.927338 4361 consensus_queue.cc:260] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.927408 4363 raft_consensus.cc:374] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:19.927470 4363 raft_consensus.cc:729] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c555fbef7f4b4d3b89a611bd03e2c5b5, State: Initialized, Role: FOLLOWER
I20260924 19:30:19.927613 4361 ts_tablet_manager.cc:1428] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:30:19.927604 4363 consensus_queue.cc:260] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.927573 4362 raft_consensus.cc:348] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.927733 4076 heartbeater.cc:503] Master 127.18.229.190:34623 was elected leader, sending a full tablet report...
I20260924 19:30:19.927744 4362 raft_consensus.cc:374] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:19.927781 4362 raft_consensus.cc:729] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ded23534574f17bf9b676937c541cc, State: Initialized, Role: FOLLOWER
I20260924 19:30:19.927851 4363 ts_tablet_manager.cc:1428] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:19.927889 4362 consensus_queue.cc:260] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.927973 4344 heartbeater.cc:503] Master 127.18.229.190:34623 was elected leader, sending a full tablet report...
I20260924 19:30:19.928158 4362 ts_tablet_manager.cc:1428] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:19.928164 4210 heartbeater.cc:503] Master 127.18.229.190:34623 was elected leader, sending a full tablet report...
W20260924 19:30:19.982399 4211 tablet.cc:2369] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:30:19.998368 4369 raft_consensus.cc:482] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:19.998546 4369 raft_consensus.cc:504] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:19.999039 4369 leader_election.cc:288] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c555fbef7f4b4d3b89a611bd03e2c5b5 (127.18.229.131:36839), 98d13ab8499746f3aca4b0fc7655c1ba (127.18.229.129:45983)
I20260924 19:30:20.003013 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.999466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:59191 (local address 127.18.229.131:36839)
0924 19:30:19.999530 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.999532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.999613 (+ 81us) server_negotiation.cc:407] Connection header received
0924 19:30:19.999652 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.999654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.999664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.999738 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:19.999901 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.000496 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.001964 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.002525 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.002578 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.002586 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.002890 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.002942 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:30:20.003052 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.999243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45983 (local address 127.18.229.130:47451)
0924 19:30:19.999290 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.999302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.999359 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.999699 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.999699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.999701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:19.999778 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.999780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.000808 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.000809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.001854 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.001855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.002810 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.002811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.002889 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.002892 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.002942 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.002984 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:30:20.003111 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.999493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:47451 (local address 127.18.229.129:45983)
0924 19:30:19.999542 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:19.999544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:19.999552 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:19.999584 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:19.999585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:19.999603 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:19.999657 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:19.999883 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.000716 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.001946 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.002712 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.002786 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.002805 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.002971 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.003019 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:30:20.003244 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:19.999251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:36839 (local address 127.18.229.130:59191)
0924 19:30:19.999529 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:19.999540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:19.999600 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:19.999736 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:19.999736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:19.999738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:19.999794 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:19.999796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.000582 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.000583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.001851 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.001855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.002682 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.002683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.002776 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.002781 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.003112 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.003149 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260924 19:30:20.003419 4030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75139d76c13b4760970bb5c4c08b4a2e" candidate_uuid: "f1ded23534574f17bf9b676937c541cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" is_pre_election: true
I20260924 19:30:20.003612 4030 raft_consensus.cc:2454] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1ded23534574f17bf9b676937c541cc in term 0.
I20260924 19:30:20.003871 4099 leader_election.cc:302] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98d13ab8499746f3aca4b0fc7655c1ba, f1ded23534574f17bf9b676937c541cc; no voters:
I20260924 19:30:20.003880 4298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75139d76c13b4760970bb5c4c08b4a2e" candidate_uuid: "f1ded23534574f17bf9b676937c541cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" is_pre_election: true
I20260924 19:30:20.004011 4298 raft_consensus.cc:2454] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1ded23534574f17bf9b676937c541cc in term 0.
I20260924 19:30:20.004014 4369 raft_consensus.cc:2781] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:30:20.004057 4369 raft_consensus.cc:482] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:20.004088 4369 raft_consensus.cc:3037] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:20.004925 4369 raft_consensus.cc:504] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:20.005075 4369 leader_election.cc:288] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [CANDIDATE]: Term 1 election: Requested vote from peers c555fbef7f4b4d3b89a611bd03e2c5b5 (127.18.229.131:36839), 98d13ab8499746f3aca4b0fc7655c1ba (127.18.229.129:45983)
I20260924 19:30:20.005244 4298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75139d76c13b4760970bb5c4c08b4a2e" candidate_uuid: "f1ded23534574f17bf9b676937c541cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5"
I20260924 19:30:20.005309 4298 raft_consensus.cc:3037] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:20.005966 4298 raft_consensus.cc:2454] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1ded23534574f17bf9b676937c541cc in term 1.
I20260924 19:30:20.006089 4030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75139d76c13b4760970bb5c4c08b4a2e" candidate_uuid: "f1ded23534574f17bf9b676937c541cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d13ab8499746f3aca4b0fc7655c1ba"
I20260924 19:30:20.006131 4097 leader_election.cc:302] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c555fbef7f4b4d3b89a611bd03e2c5b5, f1ded23534574f17bf9b676937c541cc; no voters:
I20260924 19:30:20.006217 4369 raft_consensus.cc:2781] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:20.006171 4030 raft_consensus.cc:3037] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:20.006418 4369 raft_consensus.cc:686] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 1 LEADER]: Becoming Leader. State: Replica: f1ded23534574f17bf9b676937c541cc, State: Running, Role: LEADER
I20260924 19:30:20.006516 4369 consensus_queue.cc:237] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } }
I20260924 19:30:20.007105 4030 raft_consensus.cc:2454] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1ded23534574f17bf9b676937c541cc in term 1.
I20260924 19:30:20.007655 3884 catalog_manager.cc:5495] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc reported cstate change: term changed from 0 to 1, leader changed from <none> to f1ded23534574f17bf9b676937c541cc (127.18.229.130). New cstate: current_term: 1 leader_uuid: "f1ded23534574f17bf9b676937c541cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1ded23534574f17bf9b676937c541cc" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33809 } health_report { overall_health: HEALTHY } } }
I20260924 19:30:20.013088 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.010915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.18.229.130:33809)
0924 19:30:20.010955 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.010957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.010965 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:20.011044 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.011044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.011061 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.011117 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:20.011234 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.011698 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.012474 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.012651 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.012703 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.012829 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:20.012919 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:20.012977 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.013006 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.013021 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:30:20.014688 4298 raft_consensus.cc:1261] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 1 FOLLOWER]: Refusing update from remote peer f1ded23534574f17bf9b676937c541cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:20.014741 4030 raft_consensus.cc:1261] T 75139d76c13b4760970bb5c4c08b4a2e P 98d13ab8499746f3aca4b0fc7655c1ba [term 1 FOLLOWER]: Refusing update from remote peer f1ded23534574f17bf9b676937c541cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:20.015046 4369 consensus_queue.cc:1035] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c555fbef7f4b4d3b89a611bd03e2c5b5" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:20.015182 4371 consensus_queue.cc:1035] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d13ab8499746f3aca4b0fc7655c1ba" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:20.020814 4375 mvcc.cc:205] Tried to move back new op lower bound from 7332979589177761792 to 7332979589147484160. Current Snapshot: MvccSnapshot[applied={T|T < 7332979589177761792}]
I20260924 19:30:20.020943 3885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:20.021162 3885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:20.022178 3885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25cf72c2d0a04af1930344753d1d7984: Sending DeleteTablet for 3 replicas of tablet 75139d76c13b4760970bb5c4c08b4a2e
I20260924 19:30:20.022514 4144 tablet_service.cc:1514] Processing DeleteTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:20 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:45350
I20260924 19:30:20.022504 4278 tablet_service.cc:1514] Processing DeleteTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:20 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:33636
I20260924 19:30:20.022720 4143 tablet_service.cc:1514] Processing DeleteTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:20 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:45350
I20260924 19:30:20.022825 4383 tablet_replica.cc:330] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: stopping tablet replica
I20260924 19:30:20.022917 4383 raft_consensus.cc:2229] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:20.022982 4010 tablet_service.cc:1514] Processing DeleteTablet for tablet 75139d76c13b4760970bb5c4c08b4a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:20 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:33742
I20260924 19:30:20.023025 4383 raft_consensus.cc:2258] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:20.023406 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 3947
I20260924 19:30:20.023484 4383 ts_tablet_manager.cc:1905] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:20.022744 4384 tablet_replica.cc:330] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: stopping tablet replica
I20260924 19:30:20.025388 4384 raft_consensus.cc:2229] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:30:20.025501 4384 raft_consensus.cc:2258] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:20.025520 4383 ts_tablet_manager.cc:1918] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:20.025588 4383 log.cc:1196] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/75139d76c13b4760970bb5c4c08b4a2e
I20260924 19:30:20.025862 4383 ts_tablet_manager.cc:1939] T 75139d76c13b4760970bb5c4c08b4a2e P c555fbef7f4b4d3b89a611bd03e2c5b5: Deleting consensus metadata
I20260924 19:30:20.025955 4384 ts_tablet_manager.cc:1905] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:20.026499 3869 catalog_manager.cc:4841] TS c555fbef7f4b4d3b89a611bd03e2c5b5 (127.18.229.131:36839): tablet 75139d76c13b4760970bb5c4c08b4a2e (table test-table [id=6b92814558ad4764820073b738c664be]) successfully deleted
I20260924 19:30:20.027545 4384 ts_tablet_manager.cc:1918] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:20.027622 4384 log.cc:1196] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/75139d76c13b4760970bb5c4c08b4a2e
I20260924 19:30:20.027879 4384 ts_tablet_manager.cc:1939] T 75139d76c13b4760970bb5c4c08b4a2e P f1ded23534574f17bf9b676937c541cc: Deleting consensus metadata
I20260924 19:30:20.028467 3870 catalog_manager.cc:4841] TS f1ded23534574f17bf9b676937c541cc (127.18.229.130:33809): tablet 75139d76c13b4760970bb5c4c08b4a2e (table test-table [id=6b92814558ad4764820073b738c664be]) successfully deleted
W20260924 19:30:20.028609 3870 catalog_manager.cc:4808] TS f1ded23534574f17bf9b676937c541cc (127.18.229.130:33809): delete failed for tablet 75139d76c13b4760970bb5c4c08b4a2e because the tablet was not found. No further retry: Not found: Tablet not found: 75139d76c13b4760970bb5c4c08b4a2e
W20260924 19:30:20.030444 3871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:45983 (error 108)
I20260924 19:30:20.030695 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4081
I20260924 19:30:20.031530 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.030864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:45983 (local address 127.0.0.1:33748)
0924 19:30:20.031446 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.031488 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562}
W20260924 19:30:20.031669 3871 catalog_manager.cc:4584] TS 98d13ab8499746f3aca4b0fc7655c1ba (127.18.229.129:45983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75139d76c13b4760970bb5c4c08b4a2e: Network error: Client connection negotiation failed: client connection to 127.18.229.129:45983: connect: Connection refused (error 111)
I20260924 19:30:20.035792 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4215
I20260924 19:30:20.040802 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 3853
2026-09-24T19:30:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1963 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T19:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:20Z Disabled control of system clock
I20260924 19:30:20.069216 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:36075
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:36075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:20.151738 4390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:20.152019 4390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:20.152058 4390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:20.153286 4390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:20.153362 4390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:20.153383 4390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:20.153402 4390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:20.153434 4390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:20.154773 4390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:36075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:20.155052 4390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:20.155344 4390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:20.160382 4398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.160400 4396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.160382 4395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:20.160677 4390 server_base.cc:1034] running on GCE node
I20260924 19:30:20.160912 4390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:20.161211 4390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:20.162398 4390 hybrid_clock.cc:648] HybridClock initialized: now 1790278220162392 us; error 37 us; skew 500 ppm
I20260924 19:30:20.163622 4390 webserver.cc:460] Webserver started at http://127.18.229.190:34937/ using document root <none> and password file <none>
I20260924 19:30:20.163884 4390 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:20.163956 4390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:20.164160 4390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:20.165324 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "4412f9343cf1404b961c89fca1e51b1e"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "de297b1965a2111c6f7bcfe025781502"
server_key_iv: "6f784056f066b9062aa7745f7ab75f89"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.165733 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "4412f9343cf1404b961c89fca1e51b1e"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "de297b1965a2111c6f7bcfe025781502"
server_key_iv: "6f784056f066b9062aa7745f7ab75f89"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.167169 4390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:20.168080 4404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.168470 4390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:20.168579 4390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "4412f9343cf1404b961c89fca1e51b1e"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "de297b1965a2111c6f7bcfe025781502"
server_key_iv: "6f784056f066b9062aa7745f7ab75f89"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.168663 4390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:20.190090 4390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:20.190462 4390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:20.190630 4390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:20.194766 4390 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:36075
I20260924 19:30:20.194805 4456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:36075 every 8 connection(s)
I20260924 19:30:20.195163 4390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:20.196246 4457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:20.198514 4457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e: Bootstrap starting.
I20260924 19:30:20.199272 4457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:20.199620 4457 log.cc:824] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:20.200498 4457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e: No bootstrap required, opened a new log
I20260924 19:30:20.202105 4457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4412f9343cf1404b961c89fca1e51b1e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36075 } }
I20260924 19:30:20.202245 4457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:20.202268 4457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4412f9343cf1404b961c89fca1e51b1e, State: Initialized, Role: FOLLOWER
I20260924 19:30:20.202450 4457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4412f9343cf1404b961c89fca1e51b1e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36075 } }
I20260924 19:30:20.202512 4457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:20.202570 4457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:20.202620 4457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:20.203176 4457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4412f9343cf1404b961c89fca1e51b1e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36075 } }
I20260924 19:30:20.203291 4457 leader_election.cc:302] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4412f9343cf1404b961c89fca1e51b1e; no voters:
I20260924 19:30:20.203359 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4390
I20260924 19:30:20.203462 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:20.203480 4457 leader_election.cc:288] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:20.203552 4460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:20.203692 19350 external_mini_cluster.cc:1428] Setting key f40351334f883b364551e5ca0f523f28
I20260924 19:30:20.203730 4460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [term 1 LEADER]: Becoming Leader. State: Replica: 4412f9343cf1404b961c89fca1e51b1e, State: Running, Role: LEADER
I20260924 19:30:20.203739 4457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:20.203850 4460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4412f9343cf1404b961c89fca1e51b1e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36075 } }
I20260924 19:30:20.204233 4461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4412f9343cf1404b961c89fca1e51b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4412f9343cf1404b961c89fca1e51b1e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36075 } } }
I20260924 19:30:20.204313 4461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:20.204342 4462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4412f9343cf1404b961c89fca1e51b1e. Latest consensus state: current_term: 1 leader_uuid: "4412f9343cf1404b961c89fca1e51b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4412f9343cf1404b961c89fca1e51b1e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36075 } } }
I20260924 19:30:20.204417 4462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:20.204614 4466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:20.205156 4466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:20.207249 4466 catalog_manager.cc:1347] Generated new cluster ID: d54393c125f54741917d5b13af3ca3c5
I20260924 19:30:20.207304 4466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:20.210487 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.204249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.18.229.190:36075)
0924 19:30:20.204514 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.204523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.205806 (+ 1283us) server_negotiation.cc:407] Connection header received
0924 19:30:20.205937 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.205945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.206015 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.206163 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:20.206413 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.207657 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.208487 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.208911 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.209038 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.209777 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:20.209781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:20.209781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:20.209966 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:20.209999 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:20.210011 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:20.210084 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.210166 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.210333 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":65,"threads_started":1}
I20260924 19:30:20.211728 4466 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:20.212270 4466 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:20.216496 4466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e: Generated new TSK 0
I20260924 19:30:20.216683 4466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:20.218961 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:36075
--builtin_ntp_servers=127.18.229.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:20.322841 4481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:20.323076 4481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:20.323113 4481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:20.324319 4481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:20.324374 4481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:20.324417 4481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:20.325779 4481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:20.326035 4481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:20.326333 4481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:20.332242 4487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.332381 4489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.332228 4486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:20.332628 4481 server_base.cc:1034] running on GCE node
I20260924 19:30:20.332860 4481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:20.333140 4481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:20.334303 4481 hybrid_clock.cc:648] HybridClock initialized: now 1790278220334281 us; error 33 us; skew 500 ppm
I20260924 19:30:20.335577 4481 webserver.cc:460] Webserver started at http://127.18.229.129:41229/ using document root <none> and password file <none>
I20260924 19:30:20.335842 4481 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:20.335891 4481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:20.336021 4481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:20.337191 4481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "e358dd452c76450ea1e8d31b165e8166"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "6d8e9d83771da1513fe26b7405dd13e5"
server_key_iv: "c7c169f6014ad2915e1e1bfd69570ca9"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.337551 4481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "e358dd452c76450ea1e8d31b165e8166"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "6d8e9d83771da1513fe26b7405dd13e5"
server_key_iv: "c7c169f6014ad2915e1e1bfd69570ca9"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.338847 4481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:30:20.339618 4495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.339787 4481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:20.339893 4481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "e358dd452c76450ea1e8d31b165e8166"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "6d8e9d83771da1513fe26b7405dd13e5"
server_key_iv: "c7c169f6014ad2915e1e1bfd69570ca9"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.339970 4481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:20.357890 4481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:20.358213 4481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:20.358366 4481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:20.358608 4481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:20.358958 4481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:20.358992 4481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.359014 4481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:20.359059 4481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.365096 4481 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:39227
I20260924 19:30:20.365139 4608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:39227 every 8 connection(s)
I20260924 19:30:20.365470 4481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:30:20.367333 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4481
I20260924 19:30:20.367456 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:30:20.367660 19350 external_mini_cluster.cc:1428] Setting key 47a4b7a95d378b7b15c8415e2ff739cf
I20260924 19:30:20.368672 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:36075
--builtin_ntp_servers=127.18.229.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:30:20.371158 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.366168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:58829 (local address 127.18.229.190:36075)
0924 19:30:20.366209 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.366211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.366468 (+ 257us) server_negotiation.cc:407] Connection header received
0924 19:30:20.367023 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.367025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.367039 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.367089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:20.367903 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.368545 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.369654 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.369927 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.369986 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.370918 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:20.370919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:20.370919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:20.370923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:20.370936 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:20.370940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:20.370983 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.371083 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.371105 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:20.371542 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.366154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:36075 (local address 127.18.229.129:58829)
0924 19:30:20.366339 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.366373 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:20.366906 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:20.367178 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:20.367188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:20.367197 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:20.367767 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.367774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.368673 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.368674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.369557 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.369559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.370018 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.370019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.370189 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.370775 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:20.370776 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:20.370814 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:20.370842 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:20.371004 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:20.371010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:20.371013 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.371086 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.371197 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260924 19:30:20.372661 4609 heartbeater.cc:347] Connected to a master server at 127.18.229.190:36075
I20260924 19:30:20.372766 4609 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:20.372953 4609 heartbeater.cc:511] Master 127.18.229.190:36075 requested a full tablet report, sending...
I20260924 19:30:20.373569 4421 ts_manager.cc:194] Registered new tserver with Master: e358dd452c76450ea1e8d31b165e8166 (127.18.229.129:39227)
I20260924 19:30:20.374213 4421 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.129:58829
W20260924 19:30:20.450820 4612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:20.451040 4612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:20.451062 4612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:20.452323 4612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:20.452364 4612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:20.452390 4612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:20.453786 4612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:20.454035 4612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:20.454339 4612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:20.459333 4620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.459492 4618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.459533 4617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:20.459575 4612 server_base.cc:1034] running on GCE node
I20260924 19:30:20.459873 4612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:20.460191 4612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:20.461385 4612 hybrid_clock.cc:648] HybridClock initialized: now 1790278220461376 us; error 35 us; skew 500 ppm
I20260924 19:30:20.462656 4612 webserver.cc:460] Webserver started at http://127.18.229.130:43451/ using document root <none> and password file <none>
I20260924 19:30:20.462905 4612 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:20.462975 4612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:20.463122 4612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:20.464300 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "eb2856d4fe284dfaad89ae43c2722457"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "366fafa6e8ff1b253500d0c245732e88"
server_key_iv: "75d05cbd886a4f8a286856b7d9d2ab97"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.464660 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "eb2856d4fe284dfaad89ae43c2722457"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "366fafa6e8ff1b253500d0c245732e88"
server_key_iv: "75d05cbd886a4f8a286856b7d9d2ab97"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.466236 4612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:30:20.467072 4626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.467268 4612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:20.467372 4612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "eb2856d4fe284dfaad89ae43c2722457"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "366fafa6e8ff1b253500d0c245732e88"
server_key_iv: "75d05cbd886a4f8a286856b7d9d2ab97"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.467455 4612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:20.528184 4612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:20.528541 4612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:20.528697 4612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:20.528944 4612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:20.529291 4612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:20.529348 4612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.529387 4612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:20.529419 4612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.535351 4612 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:33909
I20260924 19:30:20.535396 4739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:33909 every 8 connection(s)
I20260924 19:30:20.535748 4612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:30:20.542173 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.536862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:34529 (local address 127.18.229.190:36075)
0924 19:30:20.536908 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.536910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.536942 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:30:20.537804 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.537805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.537817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.537870 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:20.538821 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.539469 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.540559 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.540740 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.540798 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.541672 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:20.541673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:20.541673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:20.541679 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:20.541696 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:20.541701 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:20.541934 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.542084 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.542112 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:30:20.542395 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.536599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:36075 (local address 127.18.229.130:34529)
0924 19:30:20.536795 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.536850 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:20.537665 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:20.537958 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:20.537963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:20.537973 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:20.538726 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.538735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.539562 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.539563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.540484 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.540487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.540815 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.540816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.540922 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.541523 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:20.541523 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:20.541552 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:20.541577 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:20.541972 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:20.541978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:20.541981 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.542058 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.542143 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260924 19:30:20.542862 4740 heartbeater.cc:347] Connected to a master server at 127.18.229.190:36075
I20260924 19:30:20.542963 4740 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:20.543165 4740 heartbeater.cc:511] Master 127.18.229.190:36075 requested a full tablet report, sending...
I20260924 19:30:20.543555 4421 ts_manager.cc:194] Registered new tserver with Master: eb2856d4fe284dfaad89ae43c2722457 (127.18.229.130:33909)
I20260924 19:30:20.543975 4421 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.130:34529
I20260924 19:30:20.544430 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4612
I20260924 19:30:20.544543 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:30:20.544745 19350 external_mini_cluster.cc:1428] Setting key 1c45858cc2d5310f1f2afae86f5904a2
I20260924 19:30:20.545564 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:36075
--builtin_ntp_servers=127.18.229.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:20.626401 4743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:20.626602 4743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:20.626624 4743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:20.627792 4743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:20.627833 4743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:20.627861 4743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:20.629254 4743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:36075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:20.629503 4743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:20.629825 4743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:20.634814 4748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.634835 4749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:20.634892 4751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:20.634984 4743 server_base.cc:1034] running on GCE node
I20260924 19:30:20.635222 4743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:20.635504 4743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:20.636642 4743 hybrid_clock.cc:648] HybridClock initialized: now 1790278220636626 us; error 34 us; skew 500 ppm
I20260924 19:30:20.637907 4743 webserver.cc:460] Webserver started at http://127.18.229.131:44191/ using document root <none> and password file <none>
I20260924 19:30:20.638175 4743 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:20.638245 4743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:20.638392 4743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:20.639449 4743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "e579c05b08e74617acc42b59353d0e1c"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "1598c75230df6f698dad43371494391e"
server_key_iv: "428f63b6decb53ca4025598152403d7e"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.639810 4743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "e579c05b08e74617acc42b59353d0e1c"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "1598c75230df6f698dad43371494391e"
server_key_iv: "428f63b6decb53ca4025598152403d7e"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.641263 4743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:20.642048 4757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.642241 4743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:20.642354 4743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "e579c05b08e74617acc42b59353d0e1c"
format_stamp: "Formatted at 2026-09-24 19:30:20 on dist-test-slave-r8cc"
server_key: "1598c75230df6f698dad43371494391e"
server_key_iv: "428f63b6decb53ca4025598152403d7e"
server_key_version: "encryptionkey@0"
I20260924 19:30:20.642445 4743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:20.654431 4743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:20.654731 4743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:20.654889 4743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:20.655115 4743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:20.655445 4743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:20.655501 4743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.655536 4743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:20.655562 4743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:20.661413 4743 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:46805
I20260924 19:30:20.661463 4870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:46805 every 8 connection(s)
I20260924 19:30:20.661808 4743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:30:20.666774 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.662530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:56709 (local address 127.18.229.190:36075)
0924 19:30:20.662616 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.662619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.662812 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:30:20.663340 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.663341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.663352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.663405 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:20.664036 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.664592 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.665518 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.665692 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.665739 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.666575 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:20.666575 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:20.666576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:20.666579 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:20.666589 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:20.666592 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:20.666629 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.666694 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.666715 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:30:20.667004 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.662492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:36075 (local address 127.18.229.131:56709)
0924 19:30:20.662693 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.662726 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:20.663254 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:20.663460 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:20.663464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:20.663475 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:20.663967 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.663971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.664669 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.664669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.665446 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.665448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.665760 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.665761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.665875 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.666453 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:20.666453 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:20.666481 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:20.666504 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:20.666631 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:20.666633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:20.666636 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.666681 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.666757 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260924 19:30:20.667598 4871 heartbeater.cc:347] Connected to a master server at 127.18.229.190:36075
I20260924 19:30:20.667702 4871 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:20.667884 4871 heartbeater.cc:511] Master 127.18.229.190:36075 requested a full tablet report, sending...
I20260924 19:30:20.668346 4421 ts_manager.cc:194] Registered new tserver with Master: e579c05b08e74617acc42b59353d0e1c (127.18.229.131:46805)
I20260924 19:30:20.668715 4421 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.131:56709
I20260924 19:30:20.670441 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4743
I20260924 19:30:20.670547 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:30:20.670761 19350 external_mini_cluster.cc:1428] Setting key 3fb2ed781af54543a787691d3ebe1334
I20260924 19:30:20.671665 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:30:20.675531 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.673078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.18.229.190:36075)
0924 19:30:20.673139 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.673140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.673235 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:30:20.673276 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.673276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.673288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.673335 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:20.673452 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.673903 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.674588 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.674757 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.674807 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.675350 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:20.675351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:20.675351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:20.675355 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:20.675367 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:20.675371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:20.675413 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.675458 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.675478 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:30:20.677073 4421 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:30:20.678633 4421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:30:20.684697 4880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.680685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:33909 (local address 127.0.0.1:47894)
0924 19:30:20.680975 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.680991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:20.681076 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:20.681362 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:20.681362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:20.681365 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:20.681444 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.681446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.682336 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.682337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.683429 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.683432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.684400 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.684402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.684476 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.684487 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.684559 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.684609 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260924 19:30:20.684723 4881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.680770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.18.229.130:33909)
0924 19:30:20.680946 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.680952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.681049 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:30:20.681192 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.681195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.681210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.681262 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:20.681528 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.682250 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.683537 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.684317 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.684392 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.684481 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.684583 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.684626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260924 19:30:20.684883 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.681064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.18.229.129:39227)
0924 19:30:20.681210 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.681215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.681338 (+ 123us) server_negotiation.cc:407] Connection header received
0924 19:30:20.681382 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.681384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.681398 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.681470 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:20.681640 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.682716 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.683752 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.684349 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.684438 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.684528 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.684735 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.684784 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260924 19:30:20.684960 4879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.680655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:46805 (local address 127.0.0.1:55120)
0924 19:30:20.680949 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.680988 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:20.681052 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:20.681298 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:20.681301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:20.681304 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:20.681454 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.681456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.682745 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.682746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.683777 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.683779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.684700 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.684701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.684766 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.684771 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.684848 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.684891 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260924 19:30:20.685256 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.680793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.18.229.131:46805)
0924 19:30:20.681022 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:20.681025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:20.681036 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:30:20.681149 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:20.681151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:20.681168 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:20.681277 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:20.681537 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.682653 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.683867 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.684603 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.685034 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.685112 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:20.685132 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 19:30:20.685163 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260924 19:30:20.685428 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:20.680875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39227 (local address 127.0.0.1:37346)
0924 19:30:20.681230 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:20.681239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:20.681285 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:20.681533 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:20.681534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:20.681536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:20.681567 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.681569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.682775 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.682776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.683670 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:20.683673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:20.684561 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:20.684561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:20.684613 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:20.684616 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:20.684662 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:30:20.684692 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260924 19:30:20.685997 4674 tablet_service.cc:1467] Processing CreateTablet for tablet 21b5f3e30b1340298d3588b557c7f46c (DEFAULT_TABLE table=test-table [id=e2d233bc90534055a91867c335c89a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:20.685982 4805 tablet_service.cc:1467] Processing CreateTablet for tablet 21b5f3e30b1340298d3588b557c7f46c (DEFAULT_TABLE table=test-table [id=e2d233bc90534055a91867c335c89a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:20.686050 4543 tablet_service.cc:1467] Processing CreateTablet for tablet 21b5f3e30b1340298d3588b557c7f46c (DEFAULT_TABLE table=test-table [id=e2d233bc90534055a91867c335c89a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:20.686355 4543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b5f3e30b1340298d3588b557c7f46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:20.686357 4674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b5f3e30b1340298d3588b557c7f46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:20.686357 4805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b5f3e30b1340298d3588b557c7f46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:20.688995 4885 tablet_bootstrap.cc:492] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: Bootstrap starting.
I20260924 19:30:20.689616 4887 tablet_bootstrap.cc:492] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Bootstrap starting.
I20260924 19:30:20.689736 4885 tablet_bootstrap.cc:654] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:20.690105 4885 log.cc:824] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:20.690261 4887 tablet_bootstrap.cc:654] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:20.690598 4887 log.cc:824] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:20.690914 4885 tablet_bootstrap.cc:492] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: No bootstrap required, opened a new log
I20260924 19:30:20.690938 4886 tablet_bootstrap.cc:492] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Bootstrap starting.
I20260924 19:30:20.690990 4885 ts_tablet_manager.cc:1397] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:30:20.691478 4887 tablet_bootstrap.cc:492] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: No bootstrap required, opened a new log
I20260924 19:30:20.691583 4886 tablet_bootstrap.cc:654] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:20.691622 4887 ts_tablet_manager.cc:1397] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:20.691960 4886 log.cc:824] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:20.692762 4886 tablet_bootstrap.cc:492] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: No bootstrap required, opened a new log
I20260924 19:30:20.692826 4886 ts_tablet_manager.cc:1397] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:30:20.693789 4887 raft_consensus.cc:348] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:20.693753 4885 raft_consensus.cc:348] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:20.693940 4885 raft_consensus.cc:374] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:20.693941 4887 raft_consensus.cc:374] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:20.693984 4887 raft_consensus.cc:729] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e579c05b08e74617acc42b59353d0e1c, State: Initialized, Role: FOLLOWER
I20260924 19:30:20.693984 4885 raft_consensus.cc:729] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e358dd452c76450ea1e8d31b165e8166, State: Initialized, Role: FOLLOWER
I20260924 19:30:20.694096 4887 consensus_queue.cc:260] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:20.694129 4885 consensus_queue.cc:260] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:20.694375 4887 ts_tablet_manager.cc:1428] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:20.694427 4609 heartbeater.cc:503] Master 127.18.229.190:36075 was elected leader, sending a full tablet report...
I20260924 19:30:20.694504 4871 heartbeater.cc:503] Master 127.18.229.190:36075 was elected leader, sending a full tablet report...
I20260924 19:30:20.694393 4885 ts_tablet_manager.cc:1428] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 19:30:20.695402 4886 raft_consensus.cc:348] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:20.695595 4886 raft_consensus.cc:374] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:20.695636 4886 raft_consensus.cc:729] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2856d4fe284dfaad89ae43c2722457, State: Initialized, Role: FOLLOWER
I20260924 19:30:20.695757 4886 consensus_queue.cc:260] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:20.696023 4886 ts_tablet_manager.cc:1428] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 19:30:20.696098 4740 heartbeater.cc:503] Master 127.18.229.190:36075 was elected leader, sending a full tablet report...
W20260924 19:30:20.786422 4741 tablet.cc:2369] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:30:20.866139 4610 tablet.cc:2369] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:30:20.912333 4872 tablet.cc:2369] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:30:21.024195 4892 raft_consensus.cc:482] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:21.024348 4892 raft_consensus.cc:504] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:21.024868 4892 leader_election.cc:288] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e579c05b08e74617acc42b59353d0e1c (127.18.229.131:46805), eb2856d4fe284dfaad89ae43c2722457 (127.18.229.130:33909)
I20260924 19:30:21.027842 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:21.025168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:60659 (local address 127.18.229.131:46805)
0924 19:30:21.025219 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:21.025221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:21.025318 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:30:21.025387 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:21.025388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:21.025399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:21.025458 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:21.025639 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.026155 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.027108 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.027471 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.027527 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:21.027533 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:21.027718 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:30:21.027760 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:30:21.027858 4881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:21.025234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:54365 (local address 127.18.229.130:33909)
0924 19:30:21.025314 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:21.025316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:21.025424 (+ 108us) server_negotiation.cc:407] Connection header received
0924 19:30:21.025469 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:21.025470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:21.025482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:21.025538 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:21.025726 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.026181 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.027145 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.027482 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.027581 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:21.027588 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:21.027761 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 19:30:21.027798 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:30:21.027861 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:21.025078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:46805 (local address 127.18.229.129:60659)
0924 19:30:21.025228 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:21.025240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:21.025314 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:21.025476 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:21.025477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:21.025479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:21.025560 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:21.025562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.026242 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.026243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:21.027013 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:21.027015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.027566 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.027567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:21.027645 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:21.027648 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:21.027690 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:30:21.027744 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260924 19:30:21.027872 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:21.025089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:33909 (local address 127.18.229.129:54365)
0924 19:30:21.025335 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:21.025345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:21.025411 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:21.025560 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:21.025561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:21.025562 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:21.025649 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:21.025651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.026301 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.026302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:21.027021 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:21.027022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.027597 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.027598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:21.027662 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:21.027665 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:21.027737 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:30:21.027772 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":68,"threads_started":1}
I20260924 19:30:21.028295 4825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b5f3e30b1340298d3588b557c7f46c" candidate_uuid: "e358dd452c76450ea1e8d31b165e8166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e579c05b08e74617acc42b59353d0e1c" is_pre_election: true
I20260924 19:30:21.028398 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b5f3e30b1340298d3588b557c7f46c" candidate_uuid: "e358dd452c76450ea1e8d31b165e8166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2856d4fe284dfaad89ae43c2722457" is_pre_election: true
I20260924 19:30:21.028458 4825 raft_consensus.cc:2454] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e358dd452c76450ea1e8d31b165e8166 in term 0.
I20260924 19:30:21.028519 4694 raft_consensus.cc:2454] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e358dd452c76450ea1e8d31b165e8166 in term 0.
I20260924 19:30:21.028690 4496 leader_election.cc:302] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e358dd452c76450ea1e8d31b165e8166, e579c05b08e74617acc42b59353d0e1c; no voters:
I20260924 19:30:21.028843 4892 raft_consensus.cc:2781] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:30:21.028903 4892 raft_consensus.cc:482] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:21.028926 4892 raft_consensus.cc:3037] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:21.029626 4892 raft_consensus.cc:504] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:21.029770 4892 leader_election.cc:288] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [CANDIDATE]: Term 1 election: Requested vote from peers e579c05b08e74617acc42b59353d0e1c (127.18.229.131:46805), eb2856d4fe284dfaad89ae43c2722457 (127.18.229.130:33909)
I20260924 19:30:21.029932 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b5f3e30b1340298d3588b557c7f46c" candidate_uuid: "e358dd452c76450ea1e8d31b165e8166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2856d4fe284dfaad89ae43c2722457"
I20260924 19:30:21.030014 4694 raft_consensus.cc:3037] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:21.030699 4694 raft_consensus.cc:2454] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e358dd452c76450ea1e8d31b165e8166 in term 1.
I20260924 19:30:21.030820 4825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b5f3e30b1340298d3588b557c7f46c" candidate_uuid: "e358dd452c76450ea1e8d31b165e8166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e579c05b08e74617acc42b59353d0e1c"
I20260924 19:30:21.030896 4499 leader_election.cc:302] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e358dd452c76450ea1e8d31b165e8166, eb2856d4fe284dfaad89ae43c2722457; no voters:
I20260924 19:30:21.030900 4825 raft_consensus.cc:3037] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:21.030967 4892 raft_consensus.cc:2781] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:21.031100 4892 raft_consensus.cc:686] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [term 1 LEADER]: Becoming Leader. State: Replica: e358dd452c76450ea1e8d31b165e8166, State: Running, Role: LEADER
I20260924 19:30:21.031193 4892 consensus_queue.cc:237] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } }
I20260924 19:30:21.031821 4825 raft_consensus.cc:2454] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e358dd452c76450ea1e8d31b165e8166 in term 1.
I20260924 19:30:21.032024 4420 catalog_manager.cc:5495] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 reported cstate change: term changed from 0 to 1, leader changed from <none> to e358dd452c76450ea1e8d31b165e8166 (127.18.229.129). New cstate: current_term: 1 leader_uuid: "e358dd452c76450ea1e8d31b165e8166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e358dd452c76450ea1e8d31b165e8166" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 } health_report { overall_health: UNKNOWN } } }
I20260924 19:30:21.041045 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:21.038685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.18.229.129:39227)
0924 19:30:21.038740 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:21.038742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:21.038751 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:30:21.038793 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:21.038793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:21.038811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:21.038891 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:21.039008 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.039494 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.040314 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:21.040514 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:21.040589 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:21.040742 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:21.040840 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:21.040932 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:21.040951 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 19:30:21.040970 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:30:21.042690 4825 raft_consensus.cc:1261] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 1 FOLLOWER]: Refusing update from remote peer e358dd452c76450ea1e8d31b165e8166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:21.042723 4694 raft_consensus.cc:1261] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 1 FOLLOWER]: Refusing update from remote peer e358dd452c76450ea1e8d31b165e8166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:21.043022 4892 consensus_queue.cc:1035] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e579c05b08e74617acc42b59353d0e1c" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:21.043130 4896 consensus_queue.cc:1035] T 21b5f3e30b1340298d3588b557c7f46c P e358dd452c76450ea1e8d31b165e8166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2856d4fe284dfaad89ae43c2722457" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 33909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:21.045380 4899 mvcc.cc:205] Tried to move back new op lower bound from 7332979593388298240 to 7332979593344311296. Current Snapshot: MvccSnapshot[applied={T|T < 7332979593388298240}]
I20260924 19:30:21.046190 4901 mvcc.cc:205] Tried to move back new op lower bound from 7332979593388298240 to 7332979593344311296. Current Snapshot: MvccSnapshot[applied={T|T < 7332979593388298240 or (T in {7332979593388298240})}]
I20260924 19:30:21.046316 4900 mvcc.cc:205] Tried to move back new op lower bound from 7332979593388298240 to 7332979593344311296. Current Snapshot: MvccSnapshot[applied={T|T < 7332979593388298240}]
I20260924 19:30:21.047432 4421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:21.047534 4421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:21.048226 4421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4412f9343cf1404b961c89fca1e51b1e: Sending DeleteTablet for 3 replicas of tablet 21b5f3e30b1340298d3588b557c7f46c
I20260924 19:30:21.048534 4805 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b5f3e30b1340298d3588b557c7f46c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:21 UTC) from {username='slave'} at 127.0.0.1:55120
I20260924 19:30:21.048556 4543 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b5f3e30b1340298d3588b557c7f46c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:21 UTC) from {username='slave'} at 127.0.0.1:37346
I20260924 19:30:21.048619 4674 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b5f3e30b1340298d3588b557c7f46c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:21 UTC) from {username='slave'} at 127.0.0.1:47894
I20260924 19:30:21.048818 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4481
I20260924 19:30:21.048928 4911 tablet_replica.cc:330] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: stopping tablet replica
I20260924 19:30:21.049015 4911 raft_consensus.cc:2229] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:21.049099 4911 raft_consensus.cc:2258] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:21.049579 4910 tablet_replica.cc:330] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: stopping tablet replica
I20260924 19:30:21.049639 4911 ts_tablet_manager.cc:1905] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:21.049664 4910 raft_consensus.cc:2229] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:21.049748 4910 raft_consensus.cc:2258] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:21.050129 4910 ts_tablet_manager.cc:1905] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:21.051333 4911 ts_tablet_manager.cc:1918] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:21.051406 4911 log.cc:1196] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/21b5f3e30b1340298d3588b557c7f46c
I20260924 19:30:21.051676 4911 ts_tablet_manager.cc:1939] T 21b5f3e30b1340298d3588b557c7f46c P eb2856d4fe284dfaad89ae43c2722457: Deleting consensus metadata
I20260924 19:30:21.051728 4910 ts_tablet_manager.cc:1918] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:21.051793 4910 log.cc:1196] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/21b5f3e30b1340298d3588b557c7f46c
I20260924 19:30:21.052030 4910 ts_tablet_manager.cc:1939] T 21b5f3e30b1340298d3588b557c7f46c P e579c05b08e74617acc42b59353d0e1c: Deleting consensus metadata
I20260924 19:30:21.052147 4408 catalog_manager.cc:4841] TS eb2856d4fe284dfaad89ae43c2722457 (127.18.229.130:33909): tablet 21b5f3e30b1340298d3588b557c7f46c (table test-table [id=e2d233bc90534055a91867c335c89a17]) successfully deleted
I20260924 19:30:21.052459 4405 catalog_manager.cc:4841] TS e579c05b08e74617acc42b59353d0e1c (127.18.229.131:46805): tablet 21b5f3e30b1340298d3588b557c7f46c (table test-table [id=e2d233bc90534055a91867c335c89a17]) successfully deleted
W20260924 19:30:21.055307 4405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:39227 (error 108)
I20260924 19:30:21.055541 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4612
I20260924 19:30:21.055797 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:21.055674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39227 (local address 127.0.0.1:37360)
0924 19:30:21.055730 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:21.055764 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:30:21.055887 4405 catalog_manager.cc:4584] TS e358dd452c76450ea1e8d31b165e8166 (127.18.229.129:39227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21b5f3e30b1340298d3588b557c7f46c: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39227: connect: Connection refused (error 111)
I20260924 19:30:21.060576 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4743
I20260924 19:30:21.064945 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4390
2026-09-24T19:30:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1023 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:21 dist-test-slave-r8cc krb5kdc[4915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:21 dist-test-slave-r8cc krb5kdc[4915](info): set up 2 sockets
Sep 24 19:30:21 dist-test-slave-r8cc krb5kdc[4915](info): commencing operation
krb5kdc: starting...
W20260924 19:30:22.080019 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:22.194383 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:46045
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:22.274804 4928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:22.275022 4928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:22.275045 4928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:22.276260 4928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:22.276335 4928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:22.276393 4928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:22.276412 4928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:22.276435 4928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:22.277951 4928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:22.278224 4928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:22.278482 4928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:22.283360 4936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:22.283398 4928 server_base.cc:1034] running on GCE node
W20260924 19:30:22.283440 4934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:22.283368 4933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:22.283864 4928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:22.284220 4928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:22.285384 4928 hybrid_clock.cc:648] HybridClock initialized: now 1790278222285382 us; error 43 us; skew 500 ppm
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:22.287969 4928 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:22.288480 4928 webserver.cc:460] Webserver started at http://127.18.229.190:32815/ using document root <none> and password file <none>
I20260924 19:30:22.288719 4928 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:22.288795 4928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:22.288930 4928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:22.289997 4928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "c085d5a7feb048ba944925aff2e1514e"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "09cc3853be13ed0c19ec87021ec7b2d7"
server_key_iv: "2000747291af9cc6d114661429571047"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.290364 4928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "c085d5a7feb048ba944925aff2e1514e"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "09cc3853be13ed0c19ec87021ec7b2d7"
server_key_iv: "2000747291af9cc6d114661429571047"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.291693 4928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:22.292485 4943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.292658 4928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.292758 4928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "c085d5a7feb048ba944925aff2e1514e"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "09cc3853be13ed0c19ec87021ec7b2d7"
server_key_iv: "2000747291af9cc6d114661429571047"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.292852 4928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:22.315117 4928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:22.317112 4928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:22.317278 4928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:22.321262 4928 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:46045
I20260924 19:30:22.321303 4995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:46045 every 8 connection(s)
I20260924 19:30:22.321684 4928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:22.322554 4996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:22.324880 4996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e: Bootstrap starting.
I20260924 19:30:22.325469 4996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:22.325760 4996 log.cc:824] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:22.326437 4996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e: No bootstrap required, opened a new log
I20260924 19:30:22.328012 4996 raft_consensus.cc:348] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c085d5a7feb048ba944925aff2e1514e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46045 } }
I20260924 19:30:22.328171 4996 raft_consensus.cc:374] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:22.328195 4996 raft_consensus.cc:729] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c085d5a7feb048ba944925aff2e1514e, State: Initialized, Role: FOLLOWER
I20260924 19:30:22.328366 4996 consensus_queue.cc:260] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c085d5a7feb048ba944925aff2e1514e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46045 } }
I20260924 19:30:22.328431 4996 raft_consensus.cc:388] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:22.328490 4996 raft_consensus.cc:482] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:22.328540 4996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:22.329078 4996 raft_consensus.cc:504] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c085d5a7feb048ba944925aff2e1514e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46045 } }
I20260924 19:30:22.329183 4996 leader_election.cc:302] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c085d5a7feb048ba944925aff2e1514e; no voters:
I20260924 19:30:22.329190 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 4928
I20260924 19:30:22.329310 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:22.329356 4996 leader_election.cc:288] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:22.329378 4999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:22.329524 19350 external_mini_cluster.cc:1428] Setting key 23e612799439c72633c6ad2834ed98fd
I20260924 19:30:22.329547 4999 raft_consensus.cc:686] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [term 1 LEADER]: Becoming Leader. State: Replica: c085d5a7feb048ba944925aff2e1514e, State: Running, Role: LEADER
I20260924 19:30:22.329658 4996 sys_catalog.cc:564] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:22.329630 4999 consensus_queue.cc:237] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c085d5a7feb048ba944925aff2e1514e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46045 } }
I20260924 19:30:22.330370 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c085d5a7feb048ba944925aff2e1514e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c085d5a7feb048ba944925aff2e1514e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46045 } } }
I20260924 19:30:22.330464 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:22.330648 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c085d5a7feb048ba944925aff2e1514e. Latest consensus state: current_term: 1 leader_uuid: "c085d5a7feb048ba944925aff2e1514e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c085d5a7feb048ba944925aff2e1514e" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 46045 } } }
I20260924 19:30:22.330734 5010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:22.330799 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:22.331736 5010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:22.333189 5010 catalog_manager.cc:1347] Generated new cluster ID: 5356ea2ba01d44de85f2cd8e0021001a
I20260924 19:30:22.333238 5010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:22.342232 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.330009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.18.229.190:46045)
0924 19:30:22.330345 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.330356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.330399 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:30:22.330517 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.330525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.330582 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.331597 (+ 1015us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:22.331726 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.333125 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.334220 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.334865 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.334979 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.338661 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:22.338665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:22.338665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:22.338676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:22.340610 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.341015 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.341017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.341018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.341125 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.341337 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.341338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.341338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.341623 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:22.341772 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.341903 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.342026 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":70,"threads_started":1}
I20260924 19:30:22.355312 5010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:22.355873 5010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:22.358680 5010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e: Generated new TSK 0
I20260924 19:30:22.358889 5010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:30:22.386763 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:46045
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:22.480147 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:22.480338 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:22.480360 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:22.481583 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:22.481624 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:22.481650 5022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:22.483125 5022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:22.483371 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:22.483616 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:22.488632 5030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:22.488622 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:22.488619 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:22.489030 5022 server_base.cc:1034] running on GCE node
I20260924 19:30:22.489259 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:22.489531 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:22.490690 5022 hybrid_clock.cc:648] HybridClock initialized: now 1790278222490685 us; error 35 us; skew 500 ppm
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:22.493208 5022 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:30:22.493638 5022 webserver.cc:460] Webserver started at http://127.18.229.129:43169/ using document root <none> and password file <none>
I20260924 19:30:22.493877 5022 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:22.493945 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:22.494094 5022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:22.495211 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "9748622b46f14e99934ece83b62967b1"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "fcbe531fe16914d3765fb3e08b3bcb88"
server_key_iv: "b77403ab70c0dbbc3d88115305dd3d51"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.495582 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "9748622b46f14e99934ece83b62967b1"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "fcbe531fe16914d3765fb3e08b3bcb88"
server_key_iv: "b77403ab70c0dbbc3d88115305dd3d51"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.497046 5022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:22.497722 5037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.497895 5022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:22.497996 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "9748622b46f14e99934ece83b62967b1"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "fcbe531fe16914d3765fb3e08b3bcb88"
server_key_iv: "b77403ab70c0dbbc3d88115305dd3d51"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.498071 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:22.525007 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:22.526860 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:22.527047 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:22.527295 5022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:22.527616 5022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:22.527669 5022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.527710 5022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:22.527756 5022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.533720 5022 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:34605
I20260924 19:30:22.533768 5150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:34605 every 8 connection(s)
I20260924 19:30:22.534082 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:22.542440 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5022
I20260924 19:30:22.542551 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:30:22.542804 19350 external_mini_cluster.cc:1428] Setting key d6947935cb433ef95c7599caa111e1a2
I20260924 19:30:22.543761 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.534752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:34915 (local address 127.18.229.190:46045)
0924 19:30:22.534822 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.534824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.534994 (+ 170us) server_negotiation.cc:407] Connection header received
0924 19:30:22.535535 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.535536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.535548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.535607 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:22.536577 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.537076 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.538118 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.538458 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.538552 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.540890 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:22.540891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:22.540891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:22.540897 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:22.542298 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.542778 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.542779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.542779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.542835 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.543112 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.543113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.543117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.543222 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:22.543293 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.543583 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.543674 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:30:22.543949 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.534711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:46045 (local address 127.18.229.129:34915)
0924 19:30:22.534871 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:22.534914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:22.535444 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:22.535661 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:22.535664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:22.536299 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:22.536495 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.536501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.537156 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.537157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.538012 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.538014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.538574 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.538575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.538733 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.539383 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:22.539391 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:22.540803 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:22.542379 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:22.542386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:22.542389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:22.542702 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:22.542906 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:22.542907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:22.542908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:22.543028 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:22.543304 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:22.543311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:22.543446 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:30:22.543551 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:30:22.543711 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260924 19:30:22.544463 5151 heartbeater.cc:347] Connected to a master server at 127.18.229.190:46045
I20260924 19:30:22.544574 5151 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:22.544777 5151 heartbeater.cc:511] Master 127.18.229.190:46045 requested a full tablet report, sending...
I20260924 19:30:22.545305 4960 ts_manager.cc:194] Registered new tserver with Master: 9748622b46f14e99934ece83b62967b1 (127.18.229.129:34605)
I20260924 19:30:22.545851 4960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:34915
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:30:22.566478 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:46045
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:22.645773 5156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:22.646013 5156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:22.646049 5156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:22.647130 5156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:22.647194 5156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:22.647238 5156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:22.648671 5156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:22.648918 5156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:22.649156 5156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:22.654045 5162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:22.654073 5164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:22.654243 5156 server_base.cc:1034] running on GCE node
W20260924 19:30:22.654363 5161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:22.654594 5156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:22.654814 5156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:22.655956 5156 hybrid_clock.cc:648] HybridClock initialized: now 1790278222655943 us; error 27 us; skew 500 ppm
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:22.658471 5156 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:30:22.658917 5156 webserver.cc:460] Webserver started at http://127.18.229.130:38557/ using document root <none> and password file <none>
I20260924 19:30:22.659140 5156 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:22.659205 5156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:22.659351 5156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:22.660442 5156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "77981c2577fb451d83f91868b5875bc8"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "3d6c948f6e3aa0016040ffbd85203eb2"
server_key_iv: "7ca1f26c9d7a2097257aa1ce52415533"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.660808 5156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "77981c2577fb451d83f91868b5875bc8"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "3d6c948f6e3aa0016040ffbd85203eb2"
server_key_iv: "7ca1f26c9d7a2097257aa1ce52415533"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.662110 5156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:22.662824 5171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.662994 5156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:22.663096 5156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "77981c2577fb451d83f91868b5875bc8"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "3d6c948f6e3aa0016040ffbd85203eb2"
server_key_iv: "7ca1f26c9d7a2097257aa1ce52415533"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.663169 5156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:22.678146 5156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:22.680575 5156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:22.680749 5156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:22.680975 5156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:22.681275 5156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:22.681330 5156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.681370 5156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:22.681396 5156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.687209 5156 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:34307
I20260924 19:30:22.687247 5284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:34307 every 8 connection(s)
I20260924 19:30:22.687574 5156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:30:22.691195 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5156
I20260924 19:30:22.691329 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:30:22.691555 19350 external_mini_cluster.cc:1428] Setting key 1746bea544108a2b4a6ad597af0a1498
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:22.697904 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.688281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:46045 (local address 127.18.229.130:57303)
0924 19:30:22.688412 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:22.688460 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:22.688963 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:22.689227 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:22.689231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:22.689800 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:22.689988 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.689993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.690696 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.690697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.691893 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.691896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.692382 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.692384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.692510 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.693042 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:22.693050 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:22.694669 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:22.696581 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:22.696587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:22.696590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:22.696813 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:22.697031 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:22.697031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:22.697032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:22.697126 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:22.697421 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:22.697425 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:22.697537 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:30:22.697626 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:30:22.697743 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260924 19:30:22.697901 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.688319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57303 (local address 127.18.229.190:46045)
0924 19:30:22.688366 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.688368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.688540 (+ 172us) server_negotiation.cc:407] Connection header received
0924 19:30:22.689070 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.689072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.689085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.689155 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:22.690087 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.690600 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.691972 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.692287 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.692373 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.694781 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:22.694783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:22.694783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:22.694791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:22.696476 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.696892 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.696894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.696894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.696960 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.697190 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.697191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.697191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.697353 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:22.697431 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.697672 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.697766 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:30:22.698547 5285 heartbeater.cc:347] Connected to a master server at 127.18.229.190:46045
I20260924 19:30:22.698658 5285 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:22.698858 5285 heartbeater.cc:511] Master 127.18.229.190:46045 requested a full tablet report, sending...
I20260924 19:30:22.699273 4960 ts_manager.cc:194] Registered new tserver with Master: 77981c2577fb451d83f91868b5875bc8 (127.18.229.130:34307)
I20260924 19:30:22.699648 4960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:57303
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:30:22.716641 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:46045
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:22.797582 5290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:22.797775 5290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:22.797822 5290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:22.798943 5290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:22.798975 5290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:22.799001 5290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:22.800422 5290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:22.800643 5290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:22.800874 5290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:22.805673 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:22.805673 5295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:22.805717 5296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:22.805871 5290 server_base.cc:1034] running on GCE node
I20260924 19:30:22.806115 5290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:22.806347 5290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:22.807483 5290 hybrid_clock.cc:648] HybridClock initialized: now 1790278222807458 us; error 42 us; skew 500 ppm
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:22.810001 5290 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:30:22.810451 5290 webserver.cc:460] Webserver started at http://127.18.229.131:44947/ using document root <none> and password file <none>
I20260924 19:30:22.810737 5290 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:22.810823 5290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:22.810979 5290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:22.812031 5290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "6ef22c357727482a9882494c8daa31ae"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "44305a9f0fef13cab8344330c0a2a001"
server_key_iv: "a826c8dfdb9b54237c0021eac5c7eaaf"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.812458 5290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "6ef22c357727482a9882494c8daa31ae"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "44305a9f0fef13cab8344330c0a2a001"
server_key_iv: "a826c8dfdb9b54237c0021eac5c7eaaf"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.813755 5290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:22.814440 5305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.814613 5290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:22.814706 5290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "6ef22c357727482a9882494c8daa31ae"
format_stamp: "Formatted at 2026-09-24 19:30:22 on dist-test-slave-r8cc"
server_key: "44305a9f0fef13cab8344330c0a2a001"
server_key_iv: "a826c8dfdb9b54237c0021eac5c7eaaf"
server_key_version: "encryptionkey@0"
I20260924 19:30:22.814796 5290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:22.825050 5290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:22.827361 5290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:22.827519 5290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:22.827746 5290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:22.828054 5290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:22.828106 5290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.828178 5290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:22.828208 5290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:22.834287 5290 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:35427
I20260924 19:30:22.834343 5418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:35427 every 8 connection(s)
I20260924 19:30:22.834637 5290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:30:22 dist-test-slave-r8cc krb5kdc[4915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:22.841312 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5290
I20260924 19:30:22.841461 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:30:22.841658 19350 external_mini_cluster.cc:1428] Setting key 6e1a70b525c539e0921e691aea888a2b
I20260924 19:30:22.843750 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.835349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:60403 (local address 127.18.229.190:46045)
0924 19:30:22.835405 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.835406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.835623 (+ 217us) server_negotiation.cc:407] Connection header received
0924 19:30:22.836153 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.836155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.836167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.836212 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:22.837205 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.837702 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.838698 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.838899 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.838989 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.840992 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:22.840993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:22.840993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:22.841001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:22.842678 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.843024 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.843025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.843025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.843076 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.843243 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.843244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.843244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.843341 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:22.843390 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.843625 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.843687 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:30:22.843945 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.835337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:46045 (local address 127.18.229.131:60403)
0924 19:30:22.835504 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:22.835537 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:22.836027 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:22.836304 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:22.836307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:22.836923 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:22.837102 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.837106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.837794 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.837795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.838594 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.838596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.839012 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.839013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.839128 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.839547 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:22.839554 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:22.840888 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:22.842745 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:22.842750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:22.842752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:22.842959 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:22.843115 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:22.843116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:22.843116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:22.843195 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:22.843396 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:22.843400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:22.843511 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 19:30:22.843592 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:30:22.843711 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260924 19:30:22.844429 5419 heartbeater.cc:347] Connected to a master server at 127.18.229.190:46045
I20260924 19:30:22.844528 5419 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:22.844702 5419 heartbeater.cc:511] Master 127.18.229.190:46045 requested a full tablet report, sending...
I20260924 19:30:22.845069 4960 ts_manager.cc:194] Registered new tserver with Master: 6ef22c357727482a9882494c8daa31ae (127.18.229.131:35427)
I20260924 19:30:22.845445 4960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:60403
I20260924 19:30:22.853257 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:30:22.860451 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.854601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.18.229.190:46045)
0924 19:30:22.854670 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.854672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.854729 (+ 57us) server_negotiation.cc:407] Connection header received
0924 19:30:22.854769 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.854770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.854778 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.854834 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:22.855329 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.855781 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.856529 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.856700 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.856750 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.857927 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:22.857928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:22.857929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:22.857934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:22.859374 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.859719 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.859720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.859720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.859767 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:22.859953 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:22.859954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:22.859954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:22.860067 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:22.860204 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.860298 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.860345 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 19:30:22.861922 4960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:30:22.863400 4960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:30:22.870005 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.865550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:34605 (local address 127.0.0.1:34732)
0924 19:30:22.865794 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:22.865835 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:22.865899 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:22.866220 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:22.866224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:22.866226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:22.866291 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.866293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.867590 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.867591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.868916 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.868919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.869720 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.869720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.869774 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.869792 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 19:30:22.869879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:30:22.869933 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":50,"thread_start_us":43,"threads_started":1}
I20260924 19:30:22.870016 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.865926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.18.229.129:34605)
0924 19:30:22.866073 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.866076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.866084 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:22.866114 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.866116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.866128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.866216 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:22.866356 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.867496 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.868989 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.869642 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.869718 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.869772 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.869895 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.869938 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260924 19:30:22.870417 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.865565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:35427 (local address 127.0.0.1:50460)
0924 19:30:22.866413 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:22.866422 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:22.866465 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:22.866677 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:22.866678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:22.866680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:22.866723 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.866724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.867901 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.867902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.868836 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.868839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.870209 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.870209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.870266 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.870269 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:22.870321 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:30:22.870357 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":73,"threads_started":1}
I20260924 19:30:22.870654 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.865751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.18.229.131:35427)
0924 19:30:22.865956 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.865961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.866526 (+ 565us) server_negotiation.cc:407] Connection header received
0924 19:30:22.866570 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.866572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.866589 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.866664 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:22.866834 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.867796 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.868930 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.869638 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.869718 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.869772 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.870536 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.870580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260924 19:30:22.870710 5085 tablet_service.cc:1467] Processing CreateTablet for tablet d5516e42fdf641f9aa99759ac229017c (DEFAULT_TABLE table=test-table [id=c14b902875984593a531181a1c0b69be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:22.871089 5085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5516e42fdf641f9aa99759ac229017c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:22.871349 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.865550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:34307 (local address 127.0.0.1:60078)
0924 19:30:22.866062 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:22.866071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:22.866146 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:22.866345 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:22.866346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:22.866348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:22.866400 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.866402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.867488 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.867489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.868929 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:22.868931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.871109 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.871110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:22.871189 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.871193 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:22.871245 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:30:22.871285 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":75,"thread_start_us":94,"threads_started":1}
I20260924 19:30:22.871371 5353 tablet_service.cc:1467] Processing CreateTablet for tablet d5516e42fdf641f9aa99759ac229017c (DEFAULT_TABLE table=test-table [id=c14b902875984593a531181a1c0b69be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:22.871591 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:22.865750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.18.229.130:34307)
0924 19:30:22.865884 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:22.865889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:22.866140 (+ 251us) server_negotiation.cc:407] Connection header received
0924 19:30:22.866226 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:22.866229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:22.866244 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:22.866480 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:22.866552 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.867391 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.869809 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:22.870421 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:22.870602 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:22.870665 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:22.871273 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 19:30:22.871322 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260924 19:30:22.871693 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5516e42fdf641f9aa99759ac229017c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:22.872191 5219 tablet_service.cc:1467] Processing CreateTablet for tablet d5516e42fdf641f9aa99759ac229017c (DEFAULT_TABLE table=test-table [id=c14b902875984593a531181a1c0b69be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:22.872478 5219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5516e42fdf641f9aa99759ac229017c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:22.874545 5436 tablet_bootstrap.cc:492] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: Bootstrap starting.
I20260924 19:30:22.874926 5437 tablet_bootstrap.cc:492] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Bootstrap starting.
I20260924 19:30:22.875267 5436 tablet_bootstrap.cc:654] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:22.875389 5438 tablet_bootstrap.cc:492] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: Bootstrap starting.
I20260924 19:30:22.875607 5437 tablet_bootstrap.cc:654] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:22.875891 5436 log.cc:824] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:22.876024 5438 tablet_bootstrap.cc:654] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:22.876108 5437 log.cc:824] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:22.876622 5438 log.cc:824] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:22.876703 5436 tablet_bootstrap.cc:492] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: No bootstrap required, opened a new log
I20260924 19:30:22.876757 5436 ts_tablet_manager.cc:1397] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:22.877380 5438 tablet_bootstrap.cc:492] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: No bootstrap required, opened a new log
I20260924 19:30:22.877446 5438 ts_tablet_manager.cc:1397] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:22.878783 5436 raft_consensus.cc:348] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:22.878964 5436 raft_consensus.cc:374] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:22.879004 5436 raft_consensus.cc:729] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9748622b46f14e99934ece83b62967b1, State: Initialized, Role: FOLLOWER
I20260924 19:30:22.879127 5436 consensus_queue.cc:260] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:22.879400 5436 ts_tablet_manager.cc:1428] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:22.879395 5438 raft_consensus.cc:348] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:22.879539 5151 heartbeater.cc:503] Master 127.18.229.190:46045 was elected leader, sending a full tablet report...
I20260924 19:30:22.879567 5438 raft_consensus.cc:374] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:22.879618 5438 raft_consensus.cc:729] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77981c2577fb451d83f91868b5875bc8, State: Initialized, Role: FOLLOWER
I20260924 19:30:22.879737 5438 consensus_queue.cc:260] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:22.880023 5438 ts_tablet_manager.cc:1428] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:22.880244 5285 heartbeater.cc:503] Master 127.18.229.190:46045 was elected leader, sending a full tablet report...
I20260924 19:30:22.880247 5437 tablet_bootstrap.cc:492] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: No bootstrap required, opened a new log
I20260924 19:30:22.880347 5437 ts_tablet_manager.cc:1397] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 19:30:22.882097 5437 raft_consensus.cc:348] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:22.882225 5437 raft_consensus.cc:374] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:22.882246 5437 raft_consensus.cc:729] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef22c357727482a9882494c8daa31ae, State: Initialized, Role: FOLLOWER
I20260924 19:30:22.882373 5437 consensus_queue.cc:260] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:22.882628 5437 ts_tablet_manager.cc:1428] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 19:30:22.882668 5419 heartbeater.cc:503] Master 127.18.229.190:46045 was elected leader, sending a full tablet report...
W20260924 19:30:22.938212 5286 tablet.cc:2369] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:30:23.034845 5152 tablet.cc:2369] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:30:23.054385 5443 raft_consensus.cc:482] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:23.054507 5443 raft_consensus.cc:504] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:23.054937 5443 leader_election.cc:288] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9748622b46f14e99934ece83b62967b1 (127.18.229.129:34605), 6ef22c357727482a9882494c8daa31ae (127.18.229.131:35427)
I20260924 19:30:23.058864 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.055248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:41123 (local address 127.18.229.129:34605)
0924 19:30:23.055305 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.055308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.055318 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:23.055349 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.055350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.055361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.055414 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:23.055627 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.056413 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.057954 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.058514 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.058590 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.058598 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.058753 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.058792 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:30:23.058864 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.055273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:57209 (local address 127.18.229.131:35427)
0924 19:30:23.055338 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.055340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.055382 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:30:23.055483 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.055484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.055492 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.055541 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:23.055691 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.056414 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.057955 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.058514 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.058590 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.058598 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.058753 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.058800 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:30:23.058938 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.055124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:34605 (local address 127.18.229.130:41123)
0924 19:30:23.055178 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.055190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.055240 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.055448 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.055449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.055451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:23.055547 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.055549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.056497 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.056497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.057863 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.057866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.058598 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.058599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.058678 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.058680 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.058735 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.058790 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":90}
I20260924 19:30:23.058945 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.055137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:35427 (local address 127.18.229.130:57209)
0924 19:30:23.055322 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.055331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.055414 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.055564 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.055565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.055566 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:23.055627 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.055629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.056497 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.056497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.057863 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.057866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.058598 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.058599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.058674 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.058678 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.058735 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.058797 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":113,"thread_start_us":93,"threads_started":1}
I20260924 19:30:23.059329 5105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5516e42fdf641f9aa99759ac229017c" candidate_uuid: "77981c2577fb451d83f91868b5875bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9748622b46f14e99934ece83b62967b1" is_pre_election: true
I20260924 19:30:23.059412 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5516e42fdf641f9aa99759ac229017c" candidate_uuid: "77981c2577fb451d83f91868b5875bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef22c357727482a9882494c8daa31ae" is_pre_election: true
I20260924 19:30:23.059497 5105 raft_consensus.cc:2454] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77981c2577fb451d83f91868b5875bc8 in term 0.
I20260924 19:30:23.059543 5373 raft_consensus.cc:2454] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77981c2577fb451d83f91868b5875bc8 in term 0.
I20260924 19:30:23.059726 5174 leader_election.cc:302] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77981c2577fb451d83f91868b5875bc8, 9748622b46f14e99934ece83b62967b1; no voters:
I20260924 19:30:23.059861 5443 raft_consensus.cc:2781] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:30:23.059916 5443 raft_consensus.cc:482] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:23.059957 5443 raft_consensus.cc:3037] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:23.060714 5443 raft_consensus.cc:504] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:23.060861 5443 leader_election.cc:288] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [CANDIDATE]: Term 1 election: Requested vote from peers 9748622b46f14e99934ece83b62967b1 (127.18.229.129:34605), 6ef22c357727482a9882494c8daa31ae (127.18.229.131:35427)
I20260924 19:30:23.061005 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5516e42fdf641f9aa99759ac229017c" candidate_uuid: "77981c2577fb451d83f91868b5875bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef22c357727482a9882494c8daa31ae"
I20260924 19:30:23.061023 5105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5516e42fdf641f9aa99759ac229017c" candidate_uuid: "77981c2577fb451d83f91868b5875bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9748622b46f14e99934ece83b62967b1"
I20260924 19:30:23.061086 5373 raft_consensus.cc:3037] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:23.061089 5105 raft_consensus.cc:3037] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:23.061952 5373 raft_consensus.cc:2454] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77981c2577fb451d83f91868b5875bc8 in term 1.
I20260924 19:30:23.061952 5105 raft_consensus.cc:2454] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77981c2577fb451d83f91868b5875bc8 in term 1.
I20260924 19:30:23.062119 5174 leader_election.cc:302] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77981c2577fb451d83f91868b5875bc8, 9748622b46f14e99934ece83b62967b1; no voters:
I20260924 19:30:23.062229 5443 raft_consensus.cc:2781] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:23.062377 5443 raft_consensus.cc:686] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 77981c2577fb451d83f91868b5875bc8, State: Running, Role: LEADER
I20260924 19:30:23.062479 5443 consensus_queue.cc:237] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } }
I20260924 19:30:23.063339 4959 catalog_manager.cc:5495] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77981c2577fb451d83f91868b5875bc8 (127.18.229.130). New cstate: current_term: 1 leader_uuid: "77981c2577fb451d83f91868b5875bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77981c2577fb451d83f91868b5875bc8" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 34307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 } health_report { overall_health: UNKNOWN } } }
W20260924 19:30:23.085179 5420 tablet.cc:2369] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:30:23.096611 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.094123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.18.229.130:34307)
0924 19:30:23.094185 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.094187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.094196 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:30:23.094246 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.094247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.094269 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.094363 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:23.094513 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.094986 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.095765 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.095979 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.096055 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.096234 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:23.096341 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:23.096454 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.096495 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.096507 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:30:23.098387 5373 raft_consensus.cc:1261] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 1 FOLLOWER]: Refusing update from remote peer 77981c2577fb451d83f91868b5875bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:23.098399 5105 raft_consensus.cc:1261] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 1 FOLLOWER]: Refusing update from remote peer 77981c2577fb451d83f91868b5875bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:23.098755 5446 consensus_queue.cc:1035] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef22c357727482a9882494c8daa31ae" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:23.099063 5443 consensus_queue.cc:1035] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9748622b46f14e99934ece83b62967b1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:23.102275 5449 mvcc.cc:205] Tried to move back new op lower bound from 7332979601808273408 to 7332979601664577536. Current Snapshot: MvccSnapshot[applied={T|T < 7332979601808273408 or (T in {7332979601808273408})}]
I20260924 19:30:23.103852 4959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:23.103950 4959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:23.104743 4959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c085d5a7feb048ba944925aff2e1514e: Sending DeleteTablet for 3 replicas of tablet d5516e42fdf641f9aa99759ac229017c
I20260924 19:30:23.105085 5085 tablet_service.cc:1514] Processing DeleteTablet for tablet d5516e42fdf641f9aa99759ac229017c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:23 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:34732
I20260924 19:30:23.105093 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet d5516e42fdf641f9aa99759ac229017c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:23 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:50460
I20260924 19:30:23.105309 5458 tablet_replica.cc:330] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1: stopping tablet replica
I20260924 19:30:23.105309 5219 tablet_service.cc:1514] Processing DeleteTablet for tablet d5516e42fdf641f9aa99759ac229017c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:23 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:60078
I20260924 19:30:23.105422 5458 raft_consensus.cc:2229] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:23.105532 5459 tablet_replica.cc:330] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: stopping tablet replica
I20260924 19:30:23.105597 5458 raft_consensus.cc:2258] T d5516e42fdf641f9aa99759ac229017c P 9748622b46f14e99934ece83b62967b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:23.105607 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5022
I20260924 19:30:23.105679 5459 raft_consensus.cc:2229] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:23.106315 5459 raft_consensus.cc:2258] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:23.106794 5459 ts_tablet_manager.cc:1905] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:23.108824 5459 ts_tablet_manager.cc:1918] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:23.108898 5459 log.cc:1196] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/d5516e42fdf641f9aa99759ac229017c
I20260924 19:30:23.109153 5459 ts_tablet_manager.cc:1939] T d5516e42fdf641f9aa99759ac229017c P 6ef22c357727482a9882494c8daa31ae: Deleting consensus metadata
I20260924 19:30:23.109772 4947 catalog_manager.cc:4841] TS 6ef22c357727482a9882494c8daa31ae (127.18.229.131:35427): tablet d5516e42fdf641f9aa99759ac229017c (table test-table [id=c14b902875984593a531181a1c0b69be]) successfully deleted
W20260924 19:30:23.111341 4946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:34605 (error 108)
I20260924 19:30:23.111573 5460 tablet_replica.cc:330] T d5516e42fdf641f9aa99759ac229017c P 77981c2577fb451d83f91868b5875bc8: stopping tablet replica
I20260924 19:30:23.111755 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5156
I20260924 19:30:23.111851 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.111719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:34605 (local address 127.0.0.1:34748)
0924 19:30:23.111775 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.111821 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 19:30:23.112593 4946 catalog_manager.cc:4584] TS 9748622b46f14e99934ece83b62967b1 (127.18.229.129:34605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5516e42fdf641f9aa99759ac229017c: Network error: Client connection negotiation failed: client connection to 127.18.229.129:34605: connect: Connection refused (error 111)
I20260924 19:30:23.117529 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.117394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:34307 (local address 127.0.0.1:60092)
0924 19:30:23.117447 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.117494 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:30:23.117532 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5290
I20260924 19:30:23.122043 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 4928
2026-09-24T19:30:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2058 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T19:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:23Z Disabled control of system clock
I20260924 19:30:23.149597 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37809
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:37809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:23.228440 5465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:23.228652 5465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:23.228673 5465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:23.229907 5465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:23.229949 5465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:23.229962 5465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:23.229975 5465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:23.230026 5465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:23.231333 5465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:37809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:23.231550 5465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:23.231840 5465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:23.236438 5470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:23.236485 5471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:23.236522 5465 server_base.cc:1034] running on GCE node
W20260924 19:30:23.236429 5473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:23.236846 5465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:23.237133 5465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:23.238301 5465 hybrid_clock.cc:648] HybridClock initialized: now 1790278223238281 us; error 32 us; skew 500 ppm
I20260924 19:30:23.243050 5465 webserver.cc:460] Webserver started at http://127.18.229.190:44831/ using document root <none> and password file <none>
I20260924 19:30:23.243355 5465 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:23.243417 5465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:23.243543 5465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:23.244683 5465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "3ad8b85aead248fdb2731439cf43c392"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "0a572f03d360981f8ca3b3593b230644"
server_key_iv: "ed904303c438961935026de88541b2c8"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.245044 5465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "3ad8b85aead248fdb2731439cf43c392"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "0a572f03d360981f8ca3b3593b230644"
server_key_iv: "ed904303c438961935026de88541b2c8"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.246295 5465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:23.246981 5479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.247164 5465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:23.247260 5465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "3ad8b85aead248fdb2731439cf43c392"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "0a572f03d360981f8ca3b3593b230644"
server_key_iv: "ed904303c438961935026de88541b2c8"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.247330 5465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:23.265583 5465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:23.265899 5465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:23.266052 5465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:23.269963 5465 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:37809
I20260924 19:30:23.269997 5531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:37809 every 8 connection(s)
I20260924 19:30:23.270335 5465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:23.271235 5532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:23.273535 5532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392: Bootstrap starting.
I20260924 19:30:23.273802 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5465
I20260924 19:30:23.273914 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:23.274124 19350 external_mini_cluster.cc:1428] Setting key 207d0529f94ab235a689997311092c6e
I20260924 19:30:23.274276 5532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:23.274646 5532 log.cc:824] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:23.275354 5532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392: No bootstrap required, opened a new log
I20260924 19:30:23.277217 5532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad8b85aead248fdb2731439cf43c392" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37809 } }
I20260924 19:30:23.277372 5532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:23.277418 5532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad8b85aead248fdb2731439cf43c392, State: Initialized, Role: FOLLOWER
I20260924 19:30:23.277563 5532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad8b85aead248fdb2731439cf43c392" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37809 } }
I20260924 19:30:23.277652 5532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:23.277691 5532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:23.277731 5532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:23.278250 5532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad8b85aead248fdb2731439cf43c392" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37809 } }
I20260924 19:30:23.278371 5532 leader_election.cc:302] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ad8b85aead248fdb2731439cf43c392; no voters:
I20260924 19:30:23.278548 5532 leader_election.cc:288] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:23.278621 5537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:23.278820 5532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:23.278995 5537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [term 1 LEADER]: Becoming Leader. State: Replica: 3ad8b85aead248fdb2731439cf43c392, State: Running, Role: LEADER
I20260924 19:30:23.279150 5537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad8b85aead248fdb2731439cf43c392" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37809 } }
I20260924 19:30:23.279785 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.274677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.18.229.190:37809)
0924 19:30:23.274959 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.274966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.274989 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:30:23.275380 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.275410 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.275486 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.275648 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:23.275871 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.276814 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.277676 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.278127 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.278279 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.278974 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:23.278978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:23.278978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:23.279201 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:23.279242 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:23.279257 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:23.279333 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.279543 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.279632 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260924 19:30:23.279912 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ad8b85aead248fdb2731439cf43c392. Latest consensus state: current_term: 1 leader_uuid: "3ad8b85aead248fdb2731439cf43c392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad8b85aead248fdb2731439cf43c392" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37809 } } }
I20260924 19:30:23.280007 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:23.280622 5552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:23.280602 5538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ad8b85aead248fdb2731439cf43c392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad8b85aead248fdb2731439cf43c392" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37809 } } }
I20260924 19:30:23.280720 5538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:23.281038 5552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:23.282387 5552 catalog_manager.cc:1347] Generated new cluster ID: 33f93f1c653d41c48a34ea7511003937
I20260924 19:30:23.282436 5552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:23.295143 5552 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:23.295686 5552 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:23.299500 5552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392: Generated new TSK 0
I20260924 19:30:23.299687 5552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:23.304411 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37809
--builtin_ntp_servers=127.18.229.148:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:23.407291 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:23.407579 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:23.407619 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:23.410037 5556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:23.410092 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:23.410137 5556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:23.412844 5556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:23.413102 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:23.413458 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:23.418810 5561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:23.418890 5562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:23.419098 5564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:23.419270 5556 server_base.cc:1034] running on GCE node
I20260924 19:30:23.419492 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:23.419741 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:23.420905 5556 hybrid_clock.cc:648] HybridClock initialized: now 1790278223420894 us; error 30 us; skew 500 ppm
I20260924 19:30:23.422111 5556 webserver.cc:460] Webserver started at http://127.18.229.129:45523/ using document root <none> and password file <none>
I20260924 19:30:23.422370 5556 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:23.422437 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:23.422602 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:23.423628 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "8329eb7acffa4b14b3c1e54d976de399"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "65cbb09ec2e9773f199bb2283ab05c1a"
server_key_iv: "fce4dacae5322642156998b614cf844c"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.424001 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "8329eb7acffa4b14b3c1e54d976de399"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "65cbb09ec2e9773f199bb2283ab05c1a"
server_key_iv: "fce4dacae5322642156998b614cf844c"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.425369 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:23.426131 5570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.426306 5556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:23.426401 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "8329eb7acffa4b14b3c1e54d976de399"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "65cbb09ec2e9773f199bb2283ab05c1a"
server_key_iv: "fce4dacae5322642156998b614cf844c"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.426491 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:23.451933 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:23.452320 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:23.452484 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:23.452726 5556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:23.453073 5556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:23.453127 5556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.453169 5556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:23.453195 5556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.460189 5556 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:39813
I20260924 19:30:23.460234 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:39813 every 8 connection(s)
I20260924 19:30:23.460566 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:30:23.467396 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.461339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:59093 (local address 127.18.229.190:37809)
0924 19:30:23.461414 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.461416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.461739 (+ 323us) server_negotiation.cc:407] Connection header received
0924 19:30:23.462585 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.462586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.462596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.462638 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:23.463660 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.464225 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.465893 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.466070 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.466120 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.467160 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:23.467161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:23.467161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:23.467165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:23.467178 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:23.467181 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:23.467223 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.467315 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.467337 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:30:23.467723 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.461327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37809 (local address 127.18.229.129:59093)
0924 19:30:23.461554 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.461615 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.462471 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.462743 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.462749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.462761 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:23.463565 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.463573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.464320 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.464321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.465795 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.465799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.466161 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.466163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.466353 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.466976 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:23.466978 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:23.467026 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:23.467060 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:23.467226 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:23.467230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:23.467234 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.467310 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.467434 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260924 19:30:23.468384 5684 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37809
I20260924 19:30:23.468500 5684 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:23.468677 5684 heartbeater.cc:511] Master 127.18.229.190:37809 requested a full tablet report, sending...
I20260924 19:30:23.469276 5496 ts_manager.cc:194] Registered new tserver with Master: 8329eb7acffa4b14b3c1e54d976de399 (127.18.229.129:39813)
I20260924 19:30:23.469897 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5556
I20260924 19:30:23.470012 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:30:23.470218 19350 external_mini_cluster.cc:1428] Setting key 4fe19ab4e8c35d1533b19802109a7630
I20260924 19:30:23.470322 5496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.129:59093
I20260924 19:30:23.471323 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37809
--builtin_ntp_servers=127.18.229.148:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:23.552465 5687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:23.552661 5687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:23.552683 5687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:23.553831 5687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:23.553870 5687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:23.553923 5687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:23.555318 5687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:23.555559 5687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:23.555842 5687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:23.560557 5692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:23.560583 5695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:23.560598 5693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:23.560874 5687 server_base.cc:1034] running on GCE node
I20260924 19:30:23.561065 5687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:23.561307 5687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:23.562446 5687 hybrid_clock.cc:648] HybridClock initialized: now 1790278223562437 us; error 26 us; skew 500 ppm
I20260924 19:30:23.563606 5687 webserver.cc:460] Webserver started at http://127.18.229.130:35941/ using document root <none> and password file <none>
I20260924 19:30:23.563853 5687 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:23.563898 5687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:23.564072 5687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:23.565136 5687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "46cb2197186747df8ef31bed1239bbeb"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "4d436005a4a554d6e61bce18b40c565a"
server_key_iv: "f79b6d2a16e647b15cd947af95757d11"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.565491 5687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "46cb2197186747df8ef31bed1239bbeb"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "4d436005a4a554d6e61bce18b40c565a"
server_key_iv: "f79b6d2a16e647b15cd947af95757d11"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.566740 5687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:23.567409 5701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.567593 5687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:23.567677 5687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "46cb2197186747df8ef31bed1239bbeb"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "4d436005a4a554d6e61bce18b40c565a"
server_key_iv: "f79b6d2a16e647b15cd947af95757d11"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.567749 5687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:23.584846 5687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:23.585130 5687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:23.585284 5687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:23.585506 5687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:23.585865 5687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:23.585923 5687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.585954 5687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:23.585983 5687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.591400 5687 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:45293
I20260924 19:30:23.591441 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:45293 every 8 connection(s)
I20260924 19:30:23.591765 5687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:30:23.595953 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5687
I20260924 19:30:23.596066 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:30:23.596328 19350 external_mini_cluster.cc:1428] Setting key 67694a2f8e8f7efccc31e4329e267c70
I20260924 19:30:23.596774 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.592497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:34503 (local address 127.18.229.190:37809)
0924 19:30:23.592540 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.592543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.592754 (+ 211us) server_negotiation.cc:407] Connection header received
0924 19:30:23.593203 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.593205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.593220 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.593271 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:23.593892 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.594366 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.595273 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.595445 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.595492 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.596421 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:23.596423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:23.596423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:23.596428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:23.596443 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:23.596448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:23.596524 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.596652 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.596702 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:30:23.596973 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.592478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37809 (local address 127.18.229.130:34503)
0924 19:30:23.592637 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.592668 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.593123 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.593324 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.593327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.593335 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:23.593820 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.593824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.594443 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.594443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.595201 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.595203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.595515 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.595515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.595615 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.596250 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:23.596251 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:23.596290 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:23.596318 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:23.596539 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:23.596544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:23.596547 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.596642 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.596749 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260924 19:30:23.597434 5815 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37809
I20260924 19:30:23.597538 5815 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:23.597510 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37809
--builtin_ntp_servers=127.18.229.148:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:30:23.597739 5815 heartbeater.cc:511] Master 127.18.229.190:37809 requested a full tablet report, sending...
I20260924 19:30:23.598124 5496 ts_manager.cc:194] Registered new tserver with Master: 46cb2197186747df8ef31bed1239bbeb (127.18.229.130:45293)
I20260924 19:30:23.598615 5496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.130:34503
W20260924 19:30:23.676631 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:23.676863 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:23.676900 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:23.678023 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:23.678087 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:23.678128 5818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:23.679579 5818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:37809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:23.679860 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:23.680156 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:23.684980 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:23.684986 5824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:23.685159 5818 server_base.cc:1034] running on GCE node
W20260924 19:30:23.684993 5826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:23.685402 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:23.685642 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:23.686805 5818 hybrid_clock.cc:648] HybridClock initialized: now 1790278223686800 us; error 34 us; skew 500 ppm
I20260924 19:30:23.688057 5818 webserver.cc:460] Webserver started at http://127.18.229.131:46753/ using document root <none> and password file <none>
I20260924 19:30:23.688301 5818 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:23.688369 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:23.688508 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:23.689544 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "aa8702287dee49069970eeea14123118"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "d40573c41fbba6ad29043fc0cda74122"
server_key_iv: "64644df84ad7e6a99b26d4ef499b3376"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.689886 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "aa8702287dee49069970eeea14123118"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "d40573c41fbba6ad29043fc0cda74122"
server_key_iv: "64644df84ad7e6a99b26d4ef499b3376"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.691095 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:23.691783 5832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.691974 5818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:23.692068 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "aa8702287dee49069970eeea14123118"
format_stamp: "Formatted at 2026-09-24 19:30:23 on dist-test-slave-r8cc"
server_key: "d40573c41fbba6ad29043fc0cda74122"
server_key_iv: "64644df84ad7e6a99b26d4ef499b3376"
server_key_version: "encryptionkey@0"
I20260924 19:30:23.692175 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:23.730474 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:23.730803 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:23.730957 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:23.731197 5818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:23.731542 5818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:23.731596 5818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.731631 5818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:23.731674 5818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:23.738454 5818 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:38213
I20260924 19:30:23.738933 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:30:23.740216 5945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:38213 every 8 connection(s)
I20260924 19:30:23.742669 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 5818
I20260924 19:30:23.742787 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:30:23.743012 19350 external_mini_cluster.cc:1428] Setting key fe2f59ee35918c87032e15eae78d6b08
I20260924 19:30:23.745452 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.739719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:57001 (local address 127.18.229.190:37809)
0924 19:30:23.739774 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.739785 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.740045 (+ 260us) server_negotiation.cc:407] Connection header received
0924 19:30:23.740870 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.740872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.740886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.740974 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:23.741911 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.742704 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.743993 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.744247 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.744337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.745203 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:23.745204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:23.745205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:23.745209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:23.745222 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:23.745226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:23.745285 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.745355 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.745379 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:30:23.745662 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.739724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:37809 (local address 127.18.229.131:57001)
0924 19:30:23.739925 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.739965 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.740735 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.741068 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.741072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.741082 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:23.741818 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.741825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.742845 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.742846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.743888 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.743890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.744346 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.744347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.744487 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.745056 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:23.745057 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:23.745084 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:23.745106 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:23.745280 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:23.745284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:23.745287 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.745358 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.745451 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260924 19:30:23.746100 5946 heartbeater.cc:347] Connected to a master server at 127.18.229.190:37809
I20260924 19:30:23.746184 5946 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:23.746313 5946 heartbeater.cc:511] Master 127.18.229.190:37809 requested a full tablet report, sending...
I20260924 19:30:23.746625 5496 ts_manager.cc:194] Registered new tserver with Master: aa8702287dee49069970eeea14123118 (127.18.229.131:38213)
I20260924 19:30:23.746969 5496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.131:57001
I20260924 19:30:23.754703 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:30:23.758558 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.755994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.18.229.190:37809)
0924 19:30:23.756046 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.756048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.756156 (+ 108us) server_negotiation.cc:407] Connection header received
0924 19:30:23.756231 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.756231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.756239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.756287 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:23.756433 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.756877 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.757550 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.757721 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.757771 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.758352 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:23.758353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:23.758354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:23.758357 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:23.758369 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:23.758373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:23.758443 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.758499 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.758507 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:30:23.759974 5496 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:30:23.761606 5496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:30:23.767980 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.763710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:45293 (local address 127.0.0.1:36860)
0924 19:30:23.764207 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.764216 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.764252 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.764457 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.764460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.764463 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:23.764520 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.764522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.765932 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.765934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.766750 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.766753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.767654 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.767655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.767736 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.767746 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.767855 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.767909 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":32,"thread_start_us":60,"threads_started":1}
I20260924 19:30:23.767990 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.764098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.18.229.130:45293)
0924 19:30:23.764253 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.764256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.764265 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:30:23.764316 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.764318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.764330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.764425 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:23.764604 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.765841 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.766950 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.767443 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.767528 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.767621 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.767873 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.767910 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260924 19:30:23.768443 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.763681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39813 (local address 127.0.0.1:53532)
0924 19:30:23.763978 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.764019 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.764082 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.764635 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.764636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.764638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:23.764687 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.764689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.766152 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.766153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.767269 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.767272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.768241 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.768242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.768307 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.768311 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.768338 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.768374 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":79,"threads_started":1}
I20260924 19:30:23.768715 5749 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed08fddf5994322b796394967aabd16 (DEFAULT_TABLE table=test-table [id=aca2661d26e34646b35fb70771f31be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:23.768855 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.763779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.18.229.129:39813)
0924 19:30:23.764340 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.764346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.764358 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:30:23.764395 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.764397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.764413 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.764553 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:23.764768 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.765882 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.767364 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.768147 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.768607 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.768705 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.768730 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.768764 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":105,"threads_started":1}
I20260924 19:30:23.769102 5749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed08fddf5994322b796394967aabd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:23.769402 5618 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed08fddf5994322b796394967aabd16 (DEFAULT_TABLE table=test-table [id=aca2661d26e34646b35fb70771f31be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:23.769747 5618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed08fddf5994322b796394967aabd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:23.769874 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.763730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:38213 (local address 127.0.0.1:49688)
0924 19:30:23.764334 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:23.764343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:23.764409 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:23.766008 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:23.766009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:23.766026 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:23.766066 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.766068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.767440 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.767441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.768653 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:23.768656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.769624 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.769625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:23.769684 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.769688 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:23.769766 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:30:23.769807 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":71,"threads_started":1}
I20260924 19:30:23.770166 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:23.763927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.18.229.131:38213)
0924 19:30:23.764083 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:23.764087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:23.764406 (+ 319us) server_negotiation.cc:407] Connection header received
0924 19:30:23.764512 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:23.764514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:23.764528 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:23.764577 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:23.766131 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.767223 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.768744 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:23.769439 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:23.769489 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:23.769561 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:23.769790 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 19:30:23.769875 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260924 19:30:23.770937 5880 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed08fddf5994322b796394967aabd16 (DEFAULT_TABLE table=test-table [id=aca2661d26e34646b35fb70771f31be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:23.771251 5880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed08fddf5994322b796394967aabd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:23.772691 5960 tablet_bootstrap.cc:492] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: Bootstrap starting.
I20260924 19:30:23.772691 5961 tablet_bootstrap.cc:492] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Bootstrap starting.
I20260924 19:30:23.773373 5960 tablet_bootstrap.cc:654] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:23.773373 5961 tablet_bootstrap.cc:654] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:23.773687 5960 log.cc:824] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:23.773693 5961 log.cc:824] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:23.773993 5962 tablet_bootstrap.cc:492] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Bootstrap starting.
I20260924 19:30:23.774503 5962 tablet_bootstrap.cc:654] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:23.774626 5961 tablet_bootstrap.cc:492] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: No bootstrap required, opened a new log
I20260924 19:30:23.774634 5960 tablet_bootstrap.cc:492] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: No bootstrap required, opened a new log
I20260924 19:30:23.774699 5961 ts_tablet_manager.cc:1397] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:30:23.774699 5960 ts_tablet_manager.cc:1397] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:30:23.774842 5962 log.cc:824] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:23.775519 5962 tablet_bootstrap.cc:492] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: No bootstrap required, opened a new log
I20260924 19:30:23.775599 5962 ts_tablet_manager.cc:1397] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:30:23.777309 5961 raft_consensus.cc:348] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:23.777312 5960 raft_consensus.cc:348] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:23.777475 5961 raft_consensus.cc:374] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:23.777475 5960 raft_consensus.cc:374] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:23.777513 5960 raft_consensus.cc:729] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8329eb7acffa4b14b3c1e54d976de399, State: Initialized, Role: FOLLOWER
I20260924 19:30:23.777513 5961 raft_consensus.cc:729] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46cb2197186747df8ef31bed1239bbeb, State: Initialized, Role: FOLLOWER
I20260924 19:30:23.777424 5962 raft_consensus.cc:348] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:23.777578 5962 raft_consensus.cc:374] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:23.777612 5962 raft_consensus.cc:729] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8702287dee49069970eeea14123118, State: Initialized, Role: FOLLOWER
I20260924 19:30:23.777632 5960 consensus_queue.cc:260] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:23.777686 5961 consensus_queue.cc:260] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:23.777729 5962 consensus_queue.cc:260] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:23.777887 5960 ts_tablet_manager.cc:1428] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:23.777910 5961 ts_tablet_manager.cc:1428] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:23.778009 5684 heartbeater.cc:503] Master 127.18.229.190:37809 was elected leader, sending a full tablet report...
I20260924 19:30:23.778016 5962 ts_tablet_manager.cc:1428] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:30:23.778096 5946 heartbeater.cc:503] Master 127.18.229.190:37809 was elected leader, sending a full tablet report...
I20260924 19:30:23.778304 5815 heartbeater.cc:503] Master 127.18.229.190:37809 was elected leader, sending a full tablet report...
W20260924 19:30:23.842293 5816 tablet.cc:2369] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:30:23.961297 5685 tablet.cc:2369] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:30:23.989486 5947 tablet.cc:2369] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:30:24.073596 5968 raft_consensus.cc:482] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:24.073722 5968 raft_consensus.cc:504] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:24.074163 5968 leader_election.cc:288] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8329eb7acffa4b14b3c1e54d976de399 (127.18.229.129:39813), 46cb2197186747df8ef31bed1239bbeb (127.18.229.130:45293)
I20260924 19:30:24.078054 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:24.074389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:49731 (local address 127.18.229.129:39813)
0924 19:30:24.074438 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:24.074441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:24.074451 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:24.074544 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:24.074546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:24.074558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:24.074643 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:24.074831 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.075568 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.077163 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.077698 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.077776 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:24.077786 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:24.077946 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:30:24.077984 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:30:24.078054 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:24.074431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:48559 (local address 127.18.229.130:45293)
0924 19:30:24.074489 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:24.074491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:24.074601 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:30:24.074630 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:24.074631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:24.074641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:24.074701 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:24.074830 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.075567 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.077163 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.077698 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.077783 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:24.077790 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:24.077946 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 19:30:24.077984 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:30:24.078078 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:24.074326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39813 (local address 127.18.229.131:49731)
0924 19:30:24.074401 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:24.074413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:24.074459 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:24.074669 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:24.074670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:24.074672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:24.074742 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:24.074745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.075649 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.075650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:24.077068 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:24.077072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.077790 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.077791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:24.077864 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:24.077868 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:24.077925 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:30:24.077972 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":112}
I20260924 19:30:24.078084 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:24.074360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:45293 (local address 127.18.229.131:48559)
0924 19:30:24.074527 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:24.074535 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:24.074590 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:24.074704 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:24.074704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:24.074705 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:24.074760 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:24.074762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.075649 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.075650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:24.077069 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:24.077072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.077804 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.077805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:24.077864 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:24.077869 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:30:24.077925 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:30:24.077971 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":103,"thread_start_us":94,"threads_started":1}
I20260924 19:30:24.078470 5638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed08fddf5994322b796394967aabd16" candidate_uuid: "aa8702287dee49069970eeea14123118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8329eb7acffa4b14b3c1e54d976de399" is_pre_election: true
I20260924 19:30:24.078477 5769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed08fddf5994322b796394967aabd16" candidate_uuid: "aa8702287dee49069970eeea14123118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46cb2197186747df8ef31bed1239bbeb" is_pre_election: true
I20260924 19:30:24.078630 5638 raft_consensus.cc:2454] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa8702287dee49069970eeea14123118 in term 0.
I20260924 19:30:24.078630 5769 raft_consensus.cc:2454] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa8702287dee49069970eeea14123118 in term 0.
I20260924 19:30:24.078860 5835 leader_election.cc:302] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46cb2197186747df8ef31bed1239bbeb, aa8702287dee49069970eeea14123118; no voters:
I20260924 19:30:24.078998 5968 raft_consensus.cc:2781] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:30:24.079057 5968 raft_consensus.cc:482] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:24.079075 5968 raft_consensus.cc:3037] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:24.079954 5968 raft_consensus.cc:504] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:24.080096 5968 leader_election.cc:288] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [CANDIDATE]: Term 1 election: Requested vote from peers 8329eb7acffa4b14b3c1e54d976de399 (127.18.229.129:39813), 46cb2197186747df8ef31bed1239bbeb (127.18.229.130:45293)
I20260924 19:30:24.080238 5638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed08fddf5994322b796394967aabd16" candidate_uuid: "aa8702287dee49069970eeea14123118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8329eb7acffa4b14b3c1e54d976de399"
I20260924 19:30:24.080255 5769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed08fddf5994322b796394967aabd16" candidate_uuid: "aa8702287dee49069970eeea14123118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46cb2197186747df8ef31bed1239bbeb"
I20260924 19:30:24.080329 5769 raft_consensus.cc:3037] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:24.080329 5638 raft_consensus.cc:3037] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:24.081244 5769 raft_consensus.cc:2454] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa8702287dee49069970eeea14123118 in term 1.
I20260924 19:30:24.081244 5638 raft_consensus.cc:2454] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa8702287dee49069970eeea14123118 in term 1.
I20260924 19:30:24.081442 5835 leader_election.cc:302] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46cb2197186747df8ef31bed1239bbeb, aa8702287dee49069970eeea14123118; no voters:
I20260924 19:30:24.081547 5968 raft_consensus.cc:2781] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:24.081718 5968 raft_consensus.cc:686] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 1 LEADER]: Becoming Leader. State: Replica: aa8702287dee49069970eeea14123118, State: Running, Role: LEADER
I20260924 19:30:24.081815 5968 consensus_queue.cc:237] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } }
I20260924 19:30:24.082592 5494 catalog_manager.cc:5495] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa8702287dee49069970eeea14123118 (127.18.229.131). New cstate: current_term: 1 leader_uuid: "aa8702287dee49069970eeea14123118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8702287dee49069970eeea14123118" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 38213 } health_report { overall_health: HEALTHY } } }
I20260924 19:30:24.124313 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:24.121849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.18.229.131:38213)
0924 19:30:24.121911 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:24.121913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:24.121921 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:24.121957 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:24.121957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:24.121975 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:24.122049 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:24.122220 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.122655 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.123379 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:24.123563 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:24.123616 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:24.123746 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:24.124008 (+ 262us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:24.124196 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:24.124234 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 19:30:24.124246 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:30:24.126211 5769 raft_consensus.cc:1261] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 1 FOLLOWER]: Refusing update from remote peer aa8702287dee49069970eeea14123118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:24.126212 5638 raft_consensus.cc:1261] T 4ed08fddf5994322b796394967aabd16 P 8329eb7acffa4b14b3c1e54d976de399 [term 1 FOLLOWER]: Refusing update from remote peer aa8702287dee49069970eeea14123118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:24.126623 5968 consensus_queue.cc:1035] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46cb2197186747df8ef31bed1239bbeb" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:24.126797 5970 consensus_queue.cc:1035] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8329eb7acffa4b14b3c1e54d976de399" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:24.129343 5974 mvcc.cc:205] Tried to move back new op lower bound from 7332979606018031616 to 7332979605839708160. Current Snapshot: MvccSnapshot[applied={T|T < 7332979606018031616}]
I20260924 19:30:24.129532 5973 mvcc.cc:205] Tried to move back new op lower bound from 7332979606018031616 to 7332979605839708160. Current Snapshot: MvccSnapshot[applied={T|T < 7332979606018031616}]
I20260924 19:30:24.132220 5494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:24.132336 5494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:24.133123 5494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ad8b85aead248fdb2731439cf43c392: Sending DeleteTablet for 3 replicas of tablet 4ed08fddf5994322b796394967aabd16
I20260924 19:30:24.133473 5880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed08fddf5994322b796394967aabd16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:24 UTC) from {username='slave'} at 127.0.0.1:49688
I20260924 19:30:24.133455 5618 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed08fddf5994322b796394967aabd16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:24 UTC) from {username='slave'} at 127.0.0.1:53532
I20260924 19:30:24.133481 5749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed08fddf5994322b796394967aabd16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:24 UTC) from {username='slave'} at 127.0.0.1:36860
I20260924 19:30:24.133780 5985 tablet_replica.cc:330] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: stopping tablet replica
I20260924 19:30:24.133898 5985 raft_consensus.cc:2229] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:24.134039 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5556
I20260924 19:30:24.134048 5985 raft_consensus.cc:2258] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:24.135070 5985 ts_tablet_manager.cc:1905] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:24.137190 5985 ts_tablet_manager.cc:1918] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:24.137276 5985 log.cc:1196] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/4ed08fddf5994322b796394967aabd16
I20260924 19:30:24.137557 5985 ts_tablet_manager.cc:1939] T 4ed08fddf5994322b796394967aabd16 P 46cb2197186747df8ef31bed1239bbeb: Deleting consensus metadata
I20260924 19:30:24.138016 5482 catalog_manager.cc:4841] TS 46cb2197186747df8ef31bed1239bbeb (127.18.229.130:45293): tablet 4ed08fddf5994322b796394967aabd16 (table test-table [id=aca2661d26e34646b35fb70771f31be7]) successfully deleted
W20260924 19:30:24.139657 5481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:39813 (error 108)
I20260924 19:30:24.139972 5984 tablet_replica.cc:330] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: stopping tablet replica
I20260924 19:30:24.140079 5984 raft_consensus.cc:2229] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:30:24.140225 5984 raft_consensus.cc:2258] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:30:24.140636 5954 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:24.140071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:39813 (local address 127.0.0.1:53536)
0924 19:30:24.140174 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:24.140189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:24.140241 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:24.140557 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:30:24.140702 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5687
I20260924 19:30:24.142114 5984 ts_tablet_manager.cc:1905] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:30:24.142432 5481 catalog_manager.cc:4584] TS 8329eb7acffa4b14b3c1e54d976de399 (127.18.229.129:39813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ed08fddf5994322b796394967aabd16: Network error: Client connection negotiation failed: client connection to 127.18.229.129:39813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:30:24.143818 5984 ts_tablet_manager.cc:1918] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:24.143899 5984 log.cc:1196] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/4ed08fddf5994322b796394967aabd16
I20260924 19:30:24.144222 5984 ts_tablet_manager.cc:1939] T 4ed08fddf5994322b796394967aabd16 P aa8702287dee49069970eeea14123118: Deleting consensus metadata
I20260924 19:30:24.144654 5483 catalog_manager.cc:4841] TS aa8702287dee49069970eeea14123118 (127.18.229.131:38213): tablet 4ed08fddf5994322b796394967aabd16 (table test-table [id=aca2661d26e34646b35fb70771f31be7]) successfully deleted
I20260924 19:30:24.146433 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5818
I20260924 19:30:24.151117 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 5465
2026-09-24T19:30:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1028 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:24 dist-test-slave-r8cc krb5kdc[5989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:24 dist-test-slave-r8cc krb5kdc[5989](info): set up 2 sockets
Sep 24 19:30:24 dist-test-slave-r8cc krb5kdc[5989](info): commencing operation
krb5kdc: starting...
W20260924 19:30:25.163913 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:25.278497 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42443
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:25.359094 6002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:25.359342 6002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:25.359380 6002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:25.360605 6002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:25.360685 6002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:25.360707 6002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:25.360729 6002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:25.360756 6002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:25.362241 6002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:25.362512 6002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:25.362747 6002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:25.367686 6010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.367688 6008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.367686 6007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:25.367763 6002 server_base.cc:1034] running on GCE node
I20260924 19:30:25.368229 6002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:25.368578 6002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:25.369743 6002 hybrid_clock.cc:648] HybridClock initialized: now 1790278225369735 us; error 41 us; skew 500 ppm
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:25.372386 6002 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:25.372876 6002 webserver.cc:460] Webserver started at http://127.18.229.190:40097/ using document root <none> and password file <none>
I20260924 19:30:25.373117 6002 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:25.373185 6002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:25.373337 6002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:25.374383 6002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "6bc4aa61eb6947e3a52628d66edb52f4"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "18ea7215046fdfc2e632708c5ce91656"
server_key_iv: "4feb9e04176d38d5b382f7215c459920"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.374737 6002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "6bc4aa61eb6947e3a52628d66edb52f4"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "18ea7215046fdfc2e632708c5ce91656"
server_key_iv: "4feb9e04176d38d5b382f7215c459920"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.376052 6002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:25.376804 6017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.376969 6002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.377071 6002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "6bc4aa61eb6947e3a52628d66edb52f4"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "18ea7215046fdfc2e632708c5ce91656"
server_key_iv: "4feb9e04176d38d5b382f7215c459920"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.377153 6002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:25.386147 6002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:25.387907 6002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:25.388062 6002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:25.392006 6002 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42443
I20260924 19:30:25.392058 6069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42443 every 8 connection(s)
I20260924 19:30:25.392414 6002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:25.393013 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6002
I20260924 19:30:25.393103 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:25.393344 19350 external_mini_cluster.cc:1428] Setting key 32c0583f2e45f5e8cc185aa676c33c7c
I20260924 19:30:25.393350 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:25.396679 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4: Bootstrap starting.
I20260924 19:30:25.397347 6070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:25.397619 6070 log.cc:824] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:25.398319 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4: No bootstrap required, opened a new log
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:25.400167 6070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42443 } }
I20260924 19:30:25.400363 6070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:25.400404 6070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4aa61eb6947e3a52628d66edb52f4, State: Initialized, Role: FOLLOWER
I20260924 19:30:25.400564 6070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42443 } }
I20260924 19:30:25.400656 6070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:25.400701 6070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:25.400749 6070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:25.401471 6070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42443 } }
I20260924 19:30:25.401582 6070 leader_election.cc:302] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bc4aa61eb6947e3a52628d66edb52f4; no voters:
I20260924 19:30:25.401729 6070 leader_election.cc:288] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:25.401794 6075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:25.401960 6075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc4aa61eb6947e3a52628d66edb52f4, State: Running, Role: LEADER
I20260924 19:30:25.401978 6070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:25.402051 6075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42443 } }
I20260924 19:30:25.402355 6075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bc4aa61eb6947e3a52628d66edb52f4. Latest consensus state: current_term: 1 leader_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42443 } } }
I20260924 19:30:25.402443 6075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:25.402369 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4aa61eb6947e3a52628d66edb52f4" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42443 } } }
I20260924 19:30:25.402582 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:25.402640 6081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:25.403123 6081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:25.404434 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.393857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.18.229.190:42443)
0924 19:30:25.394047 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.394055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.394079 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:30:25.394473 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.394479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.394524 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.394635 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:25.395429 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.396372 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.397430 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.397886 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.397970 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.400247 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:25.400251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:25.400251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:25.400262 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:25.402232 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.402678 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.402681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.402682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.402803 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.403031 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.403032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.403032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.403471 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:25.403995 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.404067 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.404245 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260924 19:30:25.405212 6081 catalog_manager.cc:1347] Generated new cluster ID: bfa663adabfe44759522148d94ab878b
I20260924 19:30:25.405259 6081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:25.432526 6081 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:25.433336 6081 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:25.439401 6081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4: Generated new TSK 0
I20260924 19:30:25.439591 6081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.229.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
Entry for principal kudu/127.18.229.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab.
I20260924 19:30:25.468811 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42443
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:25.550239 6096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:25.550473 6096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:25.550498 6096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:25.551720 6096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:25.551764 6096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:25.551791 6096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:25.553361 6096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.129.keytab
--principal=kudu/127.18.229.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:25.553573 6096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:25.553791 6096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:30:25.558756 6096 server_base.cc:1034] running on GCE node
W20260924 19:30:25.558751 6104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.558729 6102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.558729 6101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:25.559228 6096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:25.559489 6096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:25.560639 6096 hybrid_clock.cc:648] HybridClock initialized: now 1790278225560619 us; error 36 us; skew 500 ppm
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:25.563170 6096 init.cc:377] Logged in from keytab as kudu/127.18.229.129@KRBTEST.COM (short username kudu)
I20260924 19:30:25.563640 6096 webserver.cc:460] Webserver started at http://127.18.229.129:34009/ using document root <none> and password file <none>
I20260924 19:30:25.563905 6096 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:25.563973 6096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:25.564159 6096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:25.565220 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "1cc0ee7b25be4225bc9e80eba5ec1641"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "bed00ab7d16dc641af9ad7003e870918"
server_key_iv: "65b22e50993193a673568651b9519eab"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.565586 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "1cc0ee7b25be4225bc9e80eba5ec1641"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "bed00ab7d16dc641af9ad7003e870918"
server_key_iv: "65b22e50993193a673568651b9519eab"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.566896 6096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:25.567631 6111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.567807 6096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:25.567899 6096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "1cc0ee7b25be4225bc9e80eba5ec1641"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "bed00ab7d16dc641af9ad7003e870918"
server_key_iv: "65b22e50993193a673568651b9519eab"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.567983 6096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:25.578238 6096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:25.580686 6096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:25.580871 6096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:25.581115 6096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:25.581435 6096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:25.581488 6096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.581521 6096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:25.581547 6096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.587519 6096 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:37931
I20260924 19:30:25.587567 6224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:37931 every 8 connection(s)
I20260924 19:30:25.587893 6096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:30:25.593932 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6096
I20260924 19:30:25.594064 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.129@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:25.594300 19350 external_mini_cluster.cc:1428] Setting key 94fa209dfb47ec6b85b0fd2a14ad2332
I20260924 19:30:25.598049 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.588652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:58201 (local address 127.18.229.190:42443)
0924 19:30:25.588697 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.588699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.588909 (+ 210us) server_negotiation.cc:407] Connection header received
0924 19:30:25.589444 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.589445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.589458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.589506 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:25.590491 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.591009 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.592052 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.592350 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.592437 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.594834 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:25.594835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:25.594835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:25.594842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:25.596478 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.597007 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.597007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.597008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.597059 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.597346 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.597346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.597346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.597450 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:25.597500 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.597871 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.597959 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:25.598255 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.588606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42443 (local address 127.18.229.129:58201)
0924 19:30:25.588750 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.588811 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.589327 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.589582 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.589586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.590185 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:25.590384 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.590389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.591112 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.591113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.591956 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.591959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.592450 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.592452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.592583 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.593084 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:25.593093 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:25.594745 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:25.596577 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:25.596586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:25.596589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:25.596926 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:25.597126 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:25.597127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:25.597127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:25.597273 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:25.597512 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:25.597518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:25.597700 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.597832 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.598003 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260924 19:30:25.598755 6225 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42443
I20260924 19:30:25.598860 6225 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:25.599068 6225 heartbeater.cc:511] Master 127.18.229.190:42443 requested a full tablet report, sending...
I20260924 19:30:25.599692 6034 ts_manager.cc:194] Registered new tserver with Master: 1cc0ee7b25be4225bc9e80eba5ec1641 (127.18.229.129:37931)
I20260924 19:30:25.600704 6034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.129@KRBTEST.COM'} at 127.18.229.129:58201
WARNING: no policy specified for kudu/127.18.229.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
Entry for principal kudu/127.18.229.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab.
I20260924 19:30:25.620671 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42443
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:25.703461 6230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:25.703715 6230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:25.703756 6230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:25.705032 6230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:25.705111 6230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:25.705156 6230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:25.706624 6230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.130.keytab
--principal=kudu/127.18.229.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:25.706897 6230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:25.707124 6230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:25.712260 6236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:25.712337 6230 server_base.cc:1034] running on GCE node
W20260924 19:30:25.712321 6238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.712260 6235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:25.712641 6230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:25.712908 6230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:25.714057 6230 hybrid_clock.cc:648] HybridClock initialized: now 1790278225714031 us; error 38 us; skew 500 ppm
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:25.716640 6230 init.cc:377] Logged in from keytab as kudu/127.18.229.130@KRBTEST.COM (short username kudu)
I20260924 19:30:25.717200 6230 webserver.cc:460] Webserver started at http://127.18.229.130:36717/ using document root <none> and password file <none>
I20260924 19:30:25.717432 6230 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:25.717478 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:25.717646 6230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:25.718729 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "ee8cb7e8c17643cea320250b12b952e1"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "dabae877749be84d1c2be251c9d51672"
server_key_iv: "1fb197be5782be691f35f14da2bc5f95"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.719094 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "ee8cb7e8c17643cea320250b12b952e1"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "dabae877749be84d1c2be251c9d51672"
server_key_iv: "1fb197be5782be691f35f14da2bc5f95"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.720523 6230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:25.721285 6245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.721468 6230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:25.721537 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "ee8cb7e8c17643cea320250b12b952e1"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "dabae877749be84d1c2be251c9d51672"
server_key_iv: "1fb197be5782be691f35f14da2bc5f95"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.721628 6230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:25.728938 6230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:25.730818 6230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:25.730976 6230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:25.731218 6230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:25.731532 6230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:25.731566 6230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.731590 6230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:25.731653 6230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.737625 6230 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:38841
I20260924 19:30:25.737673 6358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:38841 every 8 connection(s)
I20260924 19:30:25.737993 6230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.130@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:25.745607 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6230
I20260924 19:30:25.745754 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:30:25.745999 19350 external_mini_cluster.cc:1428] Setting key f090c25d5eb1c2673601c87be3ff3c58
I20260924 19:30:25.747303 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.738671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:39313 (local address 127.18.229.190:42443)
0924 19:30:25.738738 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.738740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.738936 (+ 196us) server_negotiation.cc:407] Connection header received
0924 19:30:25.739426 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.739428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.739439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.739494 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:25.740514 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.741014 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.741983 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.742168 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.742223 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.744427 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:25.744428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:25.744429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:25.744436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:25.745793 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.746260 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.746262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.746262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.746337 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.746597 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.746598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.746598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.746709 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:25.746763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.747069 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.747153 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:30:25.747318 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.738674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42443 (local address 127.18.229.130:39313)
0924 19:30:25.738832 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.738867 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.739336 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.739570 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.739573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.740210 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:25.740424 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.740430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.741105 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.741106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.741906 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.741908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.742241 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.742242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.742350 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.742905 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:25.742914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:25.744342 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:25.745892 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:25.745898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:25.745899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:25.746174 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:25.746424 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:25.746425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:25.746425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:25.746532 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:25.746775 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:25.746782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:25.746935 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.747024 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.747148 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260924 19:30:25.748042 6359 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42443
I20260924 19:30:25.748176 6359 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:25.748411 6359 heartbeater.cc:511] Master 127.18.229.190:42443 requested a full tablet report, sending...
I20260924 19:30:25.748832 6034 ts_manager.cc:194] Registered new tserver with Master: ee8cb7e8c17643cea320250b12b952e1 (127.18.229.130:38841)
I20260924 19:30:25.749365 6034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.130@KRBTEST.COM'} at 127.18.229.130:39313
WARNING: no policy specified for kudu/127.18.229.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
Entry for principal kudu/127.18.229.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab.
I20260924 19:30:25.771160 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42443
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:25.854359 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:25.854568 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:25.854593 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:25.855772 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:25.855822 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:25.855849 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:25.857342 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.131.keytab
--principal=kudu/127.18.229.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:25.857594 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:25.857841 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:25.862846 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.862879 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:25.862836 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:25.863019 6364 server_base.cc:1034] running on GCE node
I20260924 19:30:25.863281 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:25.863547 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:25.864722 6364 hybrid_clock.cc:648] HybridClock initialized: now 1790278225864708 us; error 45 us; skew 500 ppm
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:25.867266 6364 init.cc:377] Logged in from keytab as kudu/127.18.229.131@KRBTEST.COM (short username kudu)
I20260924 19:30:25.867698 6364 webserver.cc:460] Webserver started at http://127.18.229.131:43357/ using document root <none> and password file <none>
I20260924 19:30:25.867936 6364 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:25.868013 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:25.868189 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:25.869242 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "a84633b39c6a42388a20632a2400734d"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "6a1d67b4781d03aa1f88d7ba20657deb"
server_key_iv: "23b624ccc7f943a61d33965de3c1a522"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.869607 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "a84633b39c6a42388a20632a2400734d"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "6a1d67b4781d03aa1f88d7ba20657deb"
server_key_iv: "23b624ccc7f943a61d33965de3c1a522"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.870994 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:25.871827 6379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.872001 6364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:25.872090 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "a84633b39c6a42388a20632a2400734d"
format_stamp: "Formatted at 2026-09-24 19:30:25 on dist-test-slave-r8cc"
server_key: "6a1d67b4781d03aa1f88d7ba20657deb"
server_key_iv: "23b624ccc7f943a61d33965de3c1a522"
server_key_version: "encryptionkey@0"
I20260924 19:30:25.872174 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:25.890693 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:25.893052 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:25.893231 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:25.893486 6364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:25.893817 6364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:25.893872 6364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.893908 6364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:25.893936 6364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:25.899652 6364 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:43791
I20260924 19:30:25.899729 6492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:43791 every 8 connection(s)
I20260924 19:30:25.900065 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:30:25.905931 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6364
I20260924 19:30:25.906055 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:30:25.906287 19350 external_mini_cluster.cc:1428] Setting key 40374d9e5237298035a2fd900a4f57c1
Sep 24 19:30:25 dist-test-slave-r8cc krb5kdc[5989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278225, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.131@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:25.909530 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.900819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:44549 (local address 127.18.229.190:42443)
0924 19:30:25.900885 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.900887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.901091 (+ 204us) server_negotiation.cc:407] Connection header received
0924 19:30:25.901706 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.901708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.901721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.901789 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:25.902881 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.903391 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.904487 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.904701 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.904793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.907151 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:25.907153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:25.907153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:25.907161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:25.908388 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.908739 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.908741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.908741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.908801 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.909019 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.909020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.909020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.909122 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:25.909173 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.909408 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.909467 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:30:25.909754 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.900795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:42443 (local address 127.18.229.131:44549)
0924 19:30:25.900967 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.901004 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.901577 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.901864 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.901868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.902551 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:25.902779 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.902785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.903515 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.903516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.904398 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.904401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.904795 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.904796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.904922 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.905379 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:25.905387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:25.907035 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:25.908452 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:25.908458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:25.908460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:25.908669 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:25.908868 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:25.908869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:25.908869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:25.908964 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:25.909175 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:25.909179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:25.909300 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.909382 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.909497 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260924 19:30:25.910176 6493 heartbeater.cc:347] Connected to a master server at 127.18.229.190:42443
I20260924 19:30:25.910270 6493 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:25.910408 6493 heartbeater.cc:511] Master 127.18.229.190:42443 requested a full tablet report, sending...
I20260924 19:30:25.910763 6034 ts_manager.cc:194] Registered new tserver with Master: a84633b39c6a42388a20632a2400734d (127.18.229.131:43791)
I20260924 19:30:25.911096 6034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.229.131@KRBTEST.COM'} at 127.18.229.131:44549
I20260924 19:30:25.917933 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:30:25.925168 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.919226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.18.229.190:42443)
0924 19:30:25.919314 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.919317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.919414 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:30:25.919498 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.919498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.919510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.919566 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:25.920154 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.920649 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.921396 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.921604 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.921709 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.922842 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:25.922843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:25.922843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:25.922851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:25.923949 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.924331 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.924332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.924333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.924407 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:25.924587 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:25.924588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:25.924588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:25.924746 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:25.924860 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.924961 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.925041 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:30:25.926656 6034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:30:25.928270 6034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:30:25.934269 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.930287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38841 (local address 127.0.0.1:57594)
0924 19:30:25.930954 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.930961 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.931001 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.931250 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.931251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.931253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:25.931322 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.931324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.932310 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.932311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.933240 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.933242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.933969 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.933970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.934052 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.934065 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.934137 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.934193 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":71,"threads_started":1}
I20260924 19:30:25.934280 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.930399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.18.229.130:38841)
0924 19:30:25.930674 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.930678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.931007 (+ 329us) server_negotiation.cc:407] Connection header received
0924 19:30:25.931079 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.931081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.931096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.931167 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:25.931410 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.932189 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.933379 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.933879 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.933970 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.934029 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.934162 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.934201 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
I20260924 19:30:25.935197 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.930399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.18.229.129:37931)
0924 19:30:25.930599 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.930604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.930864 (+ 260us) server_negotiation.cc:407] Connection header received
0924 19:30:25.930953 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.930955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.930972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.931058 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:25.931352 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.932545 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.934030 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.934765 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.934815 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.934866 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.935063 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.935104 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260924 19:30:25.935185 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.930293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:37931 (local address 127.0.0.1:40694)
0924 19:30:25.930729 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.930773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.930861 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.931148 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.931151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.931154 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:25.931243 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.931246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.932667 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.932668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.933928 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.933931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.934929 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.934930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.934992 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.934997 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.935047 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.935088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":92,"thread_start_us":65,"threads_started":1}
I20260924 19:30:25.935091 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 28023d858996456484b1375dc3abcddd (DEFAULT_TABLE table=test-table [id=74b66198cf004011bb993a01549f591a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:25.935366 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.930612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.18.229.131:43791)
0924 19:30:25.930803 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.930807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.930851 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:30:25.930952 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.930954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.930971 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.931058 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:25.931334 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.932554 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.934026 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.934863 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.934933 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.934981 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.935230 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.935289 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260924 19:30:25.935384 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.930517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:43791 (local address 127.0.0.1:41458)
0924 19:30:25.930797 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.930806 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.930861 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.931148 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.931151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.931156 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:25.931243 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.931246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.932666 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.932667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.933928 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.933931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.934945 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.934946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.935010 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.935015 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.935071 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.935117 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":60,"thread_start_us":47,"threads_started":1}
I20260924 19:30:25.935490 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28023d858996456484b1375dc3abcddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:25.936019 6159 tablet_service.cc:1467] Processing CreateTablet for tablet 28023d858996456484b1375dc3abcddd (DEFAULT_TABLE table=test-table [id=74b66198cf004011bb993a01549f591a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:25.936383 6159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28023d858996456484b1375dc3abcddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:25.936715 6427 tablet_service.cc:1467] Processing CreateTablet for tablet 28023d858996456484b1375dc3abcddd (DEFAULT_TABLE table=test-table [id=74b66198cf004011bb993a01549f591a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:25.936947 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28023d858996456484b1375dc3abcddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:25.939028 6507 tablet_bootstrap.cc:492] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Bootstrap starting.
I20260924 19:30:25.939520 6508 tablet_bootstrap.cc:492] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Bootstrap starting.
I20260924 19:30:25.939738 6509 tablet_bootstrap.cc:492] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: Bootstrap starting.
I20260924 19:30:25.939877 6507 tablet_bootstrap.cc:654] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:25.940327 6508 tablet_bootstrap.cc:654] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:25.940512 6509 tablet_bootstrap.cc:654] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:25.940506 6507 log.cc:824] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:25.940958 6508 log.cc:824] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:25.941063 6509 log.cc:824] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:25.941332 6507 tablet_bootstrap.cc:492] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: No bootstrap required, opened a new log
I20260924 19:30:25.941423 6507 ts_tablet_manager.cc:1397] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:30:25.941713 6508 tablet_bootstrap.cc:492] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: No bootstrap required, opened a new log
I20260924 19:30:25.941731 6509 tablet_bootstrap.cc:492] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: No bootstrap required, opened a new log
I20260924 19:30:25.941787 6508 ts_tablet_manager.cc:1397] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:25.941794 6509 ts_tablet_manager.cc:1397] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:25.942883 6507 raft_consensus.cc:348] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.943051 6507 raft_consensus.cc:374] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:25.943095 6507 raft_consensus.cc:729] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8cb7e8c17643cea320250b12b952e1, State: Initialized, Role: FOLLOWER
I20260924 19:30:25.943243 6507 consensus_queue.cc:260] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.943514 6507 ts_tablet_manager.cc:1428] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:30:25.943650 6359 heartbeater.cc:503] Master 127.18.229.190:42443 was elected leader, sending a full tablet report...
I20260924 19:30:25.943822 6508 raft_consensus.cc:348] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.944000 6508 raft_consensus.cc:374] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:25.944042 6508 raft_consensus.cc:729] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84633b39c6a42388a20632a2400734d, State: Initialized, Role: FOLLOWER
I20260924 19:30:25.944185 6508 consensus_queue.cc:260] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.944149 6509 raft_consensus.cc:348] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.944300 6509 raft_consensus.cc:374] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:25.944345 6509 raft_consensus.cc:729] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc0ee7b25be4225bc9e80eba5ec1641, State: Initialized, Role: FOLLOWER
I20260924 19:30:25.944473 6508 ts_tablet_manager.cc:1428] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:25.944463 6509 consensus_queue.cc:260] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.944614 6493 heartbeater.cc:503] Master 127.18.229.190:42443 was elected leader, sending a full tablet report...
I20260924 19:30:25.944722 6509 ts_tablet_manager.cc:1428] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:25.944777 6225 heartbeater.cc:503] Master 127.18.229.190:42443 was elected leader, sending a full tablet report...
I20260924 19:30:25.945636 6515 raft_consensus.cc:482] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:25.945724 6515 raft_consensus.cc:504] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.946036 6515 leader_election.cc:288] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee8cb7e8c17643cea320250b12b952e1 (127.18.229.130:38841), a84633b39c6a42388a20632a2400734d (127.18.229.131:43791)
I20260924 19:30:25.949244 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.946269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:34279 (local address 127.18.229.130:38841)
0924 19:30:25.946311 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.946313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.946321 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:25.946434 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.946435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.946446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.946507 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:25.946683 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.947165 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.948484 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.948878 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.948971 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.948981 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.949142 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.949182 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:30:25.949245 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.946210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38841 (local address 127.18.229.129:34279)
0924 19:30:25.946268 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.946281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.946323 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.946557 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.946558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.946560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:25.946625 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.946627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.947241 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.947242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.948399 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.948402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.948971 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.948972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.949058 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.949061 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.949134 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.949175 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260924 19:30:25.949890 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.946444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:47109 (local address 127.18.229.131:43791)
0924 19:30:25.946485 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.946487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.946567 (+ 80us) server_negotiation.cc:407] Connection header received
0924 19:30:25.946615 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.946616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.946623 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.946677 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:25.947378 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.948080 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.949033 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.949534 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.949609 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.949617 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.949784 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.949831 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:30:25.949899 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.946367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:43791 (local address 127.18.229.129:47109)
0924 19:30:25.946515 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.946524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:25.946575 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:25.947244 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:25.947245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:25.947246 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:25.947302 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.947304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.948208 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.948209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.948941 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:25.948943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.949616 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.949616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:25.949697 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.949701 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:25.949778 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:30:25.949825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260924 19:30:25.949922 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28023d858996456484b1375dc3abcddd" candidate_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8cb7e8c17643cea320250b12b952e1" is_pre_election: true
I20260924 19:30:25.950089 6313 raft_consensus.cc:2454] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0ee7b25be4225bc9e80eba5ec1641 in term 0.
I20260924 19:30:25.950359 6113 leader_election.cc:302] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0ee7b25be4225bc9e80eba5ec1641, ee8cb7e8c17643cea320250b12b952e1; no voters:
I20260924 19:30:25.950440 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28023d858996456484b1375dc3abcddd" candidate_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84633b39c6a42388a20632a2400734d" is_pre_election: true
I20260924 19:30:25.950544 6515 raft_consensus.cc:2781] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:30:25.950610 6515 raft_consensus.cc:482] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:25.950598 6447 raft_consensus.cc:2454] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc0ee7b25be4225bc9e80eba5ec1641 in term 0.
I20260924 19:30:25.950642 6515 raft_consensus.cc:3037] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:25.951367 6515 raft_consensus.cc:504] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.951468 6515 leader_election.cc:288] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [CANDIDATE]: Term 1 election: Requested vote from peers ee8cb7e8c17643cea320250b12b952e1 (127.18.229.130:38841), a84633b39c6a42388a20632a2400734d (127.18.229.131:43791)
I20260924 19:30:25.951679 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28023d858996456484b1375dc3abcddd" candidate_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8cb7e8c17643cea320250b12b952e1"
I20260924 19:30:25.951696 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28023d858996456484b1375dc3abcddd" candidate_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84633b39c6a42388a20632a2400734d"
I20260924 19:30:25.951756 6313 raft_consensus.cc:3037] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:25.951794 6447 raft_consensus.cc:3037] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:25.952481 6313 raft_consensus.cc:2454] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0ee7b25be4225bc9e80eba5ec1641 in term 1.
I20260924 19:30:25.952632 6447 raft_consensus.cc:2454] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc0ee7b25be4225bc9e80eba5ec1641 in term 1.
I20260924 19:30:25.952639 6113 leader_election.cc:302] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc0ee7b25be4225bc9e80eba5ec1641, ee8cb7e8c17643cea320250b12b952e1; no voters:
I20260924 19:30:25.952721 6515 raft_consensus.cc:2781] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:25.952872 6515 raft_consensus.cc:686] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc0ee7b25be4225bc9e80eba5ec1641, State: Running, Role: LEADER
I20260924 19:30:25.952970 6515 consensus_queue.cc:237] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } }
I20260924 19:30:25.953730 6033 catalog_manager.cc:5495] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc0ee7b25be4225bc9e80eba5ec1641 (127.18.229.129). New cstate: current_term: 1 leader_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc0ee7b25be4225bc9e80eba5ec1641" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 } health_report { overall_health: UNKNOWN } } }
I20260924 19:30:25.958899 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.956859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.18.229.129:37931)
0924 19:30:25.956892 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:25.956894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:25.956952 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:30:25.957010 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:25.957011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:25.957026 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:25.957084 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:25.957194 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.957624 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.958298 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:25.958469 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:25.958516 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:25.958641 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:25.958730 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:25.958793 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:25.958822 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:30:25.958837 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:30:25.960592 6447 raft_consensus.cc:1261] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 1 FOLLOWER]: Refusing update from remote peer 1cc0ee7b25be4225bc9e80eba5ec1641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:25.960608 6313 raft_consensus.cc:1261] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 1 FOLLOWER]: Refusing update from remote peer 1cc0ee7b25be4225bc9e80eba5ec1641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:25.960948 6517 consensus_queue.cc:1035] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84633b39c6a42388a20632a2400734d" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:25.961066 6515 consensus_queue.cc:1035] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8cb7e8c17643cea320250b12b952e1" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:25.965907 6034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:25.966010 6034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:25.966703 6034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bc4aa61eb6947e3a52628d66edb52f4: Sending DeleteTablet for 3 replicas of tablet 28023d858996456484b1375dc3abcddd
I20260924 19:30:25.967067 6293 tablet_service.cc:1514] Processing DeleteTablet for tablet 28023d858996456484b1375dc3abcddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:25 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:57594
I20260924 19:30:25.967073 6427 tablet_service.cc:1514] Processing DeleteTablet for tablet 28023d858996456484b1375dc3abcddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:25 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:41458
I20260924 19:30:25.967291 6159 tablet_service.cc:1514] Processing DeleteTablet for tablet 28023d858996456484b1375dc3abcddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:25 UTC) from {username='kudu', principal='kudu/127.18.229.190@KRBTEST.COM'} at 127.0.0.1:40694
I20260924 19:30:25.967506 6532 tablet_replica.cc:330] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641: stopping tablet replica
I20260924 19:30:25.967608 6532 raft_consensus.cc:2229] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:30:25.967705 6532 raft_consensus.cc:2258] T 28023d858996456484b1375dc3abcddd P 1cc0ee7b25be4225bc9e80eba5ec1641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:25.967836 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6096
I20260924 19:30:25.968426 6531 tablet_replica.cc:330] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: stopping tablet replica
I20260924 19:30:25.968542 6531 raft_consensus.cc:2229] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:25.968649 6531 raft_consensus.cc:2258] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:25.969079 6531 ts_tablet_manager.cc:1905] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:25.970606 6531 ts_tablet_manager.cc:1918] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:25.970696 6531 log.cc:1196] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/28023d858996456484b1375dc3abcddd
I20260924 19:30:25.970957 6531 ts_tablet_manager.cc:1939] T 28023d858996456484b1375dc3abcddd P a84633b39c6a42388a20632a2400734d: Deleting consensus metadata
I20260924 19:30:25.971400 6019 catalog_manager.cc:4841] TS a84633b39c6a42388a20632a2400734d (127.18.229.131:43791): tablet 28023d858996456484b1375dc3abcddd (table test-table [id=74b66198cf004011bb993a01549f591a]) successfully deleted
I20260924 19:30:25.972239 6530 tablet_replica.cc:330] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: stopping tablet replica
I20260924 19:30:25.972373 6530 raft_consensus.cc:2229] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:25.972501 6530 raft_consensus.cc:2258] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:25.972940 6530 ts_tablet_manager.cc:1905] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:25.974246 6530 ts_tablet_manager.cc:1918] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:25.974334 6530 log.cc:1196] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790277987605586-19350-0/minicluster-data/ts-1/wal/wals/28023d858996456484b1375dc3abcddd
W20260924 19:30:25.974686 6021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:37931 (error 108)
I20260924 19:30:25.974735 6530 ts_tablet_manager.cc:1939] T 28023d858996456484b1375dc3abcddd P ee8cb7e8c17643cea320250b12b952e1: Deleting consensus metadata
I20260924 19:30:25.974870 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6230
I20260924 19:30:25.975482 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.975057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:37931 (local address 127.0.0.1:40708)
0924 19:30:25.975400 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.975448 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
W20260924 19:30:25.975595 6021 catalog_manager.cc:4584] TS 1cc0ee7b25be4225bc9e80eba5ec1641 (127.18.229.129:37931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28023d858996456484b1375dc3abcddd: Network error: Client connection negotiation failed: client connection to 127.18.229.129:37931: connect: Connection refused (error 111)
I20260924 19:30:25.980024 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6364
I20260924 19:30:25.980165 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:25.980025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:38841 (local address 127.0.0.1:57610)
0924 19:30:25.980077 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:25.980125 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:30:25.985311 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6002
2026-09-24T19:30:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1834 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T19:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:26Z Disabled control of system clock
I20260924 19:30:26.012830 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:39639
--webserver_interface=127.18.229.190
--webserver_port=0
--builtin_ntp_servers=127.18.229.148:32863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:26.093921 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:26.094146 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:26.094169 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:26.095290 6537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:26.095335 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:26.095347 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:26.095359 6537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:26.095372 6537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:26.096808 6537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:39639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:26.097002 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:26.097229 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:26.101850 6543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.102041 6537 server_base.cc:1034] running on GCE node
W20260924 19:30:26.101845 6545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:26.101845 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.102492 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:26.102757 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:26.103921 6537 hybrid_clock.cc:648] HybridClock initialized: now 1790278226103910 us; error 30 us; skew 500 ppm
I20260924 19:30:26.105194 6537 webserver.cc:460] Webserver started at http://127.18.229.190:34603/ using document root <none> and password file <none>
I20260924 19:30:26.105428 6537 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:26.105492 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:26.105643 6537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:26.106738 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "b6af9beb259f4827be6be10aba4a8be1"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "547b54244a6c7c38ed557aaf2e62fb4e"
server_key_iv: "c5e54240d62540502142ec82eadb2185"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.107092 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "b6af9beb259f4827be6be10aba4a8be1"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "547b54244a6c7c38ed557aaf2e62fb4e"
server_key_iv: "c5e54240d62540502142ec82eadb2185"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.108366 6537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:26.109040 6551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.109200 6537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.109300 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "b6af9beb259f4827be6be10aba4a8be1"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "547b54244a6c7c38ed557aaf2e62fb4e"
server_key_iv: "c5e54240d62540502142ec82eadb2185"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.109366 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:26.118603 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:26.118862 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:26.118996 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:26.122573 6537 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:39639
I20260924 19:30:26.122630 6603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:39639 every 8 connection(s)
I20260924 19:30:26.122944 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:26.123857 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:26.126042 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1: Bootstrap starting.
I20260924 19:30:26.126593 6604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:26.126888 6604 log.cc:824] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:26.127137 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6537
I20260924 19:30:26.127251 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:26.127449 19350 external_mini_cluster.cc:1428] Setting key 7e517e0e60465612c77f50850448d164
I20260924 19:30:26.127656 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1: No bootstrap required, opened a new log
I20260924 19:30:26.129570 6604 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6af9beb259f4827be6be10aba4a8be1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39639 } }
I20260924 19:30:26.129719 6604 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:26.129765 6604 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6af9beb259f4827be6be10aba4a8be1, State: Initialized, Role: FOLLOWER
I20260924 19:30:26.129900 6604 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6af9beb259f4827be6be10aba4a8be1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39639 } }
I20260924 19:30:26.130002 6604 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:26.130046 6604 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:26.130095 6604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:26.130604 6604 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6af9beb259f4827be6be10aba4a8be1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39639 } }
I20260924 19:30:26.130712 6604 leader_election.cc:302] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6af9beb259f4827be6be10aba4a8be1; no voters:
I20260924 19:30:26.130894 6604 leader_election.cc:288] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:26.130928 6609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:26.131059 6609 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [term 1 LEADER]: Becoming Leader. State: Replica: b6af9beb259f4827be6be10aba4a8be1, State: Running, Role: LEADER
I20260924 19:30:26.131139 6609 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6af9beb259f4827be6be10aba4a8be1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39639 } }
I20260924 19:30:26.131146 6604 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:26.131477 6609 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6af9beb259f4827be6be10aba4a8be1. Latest consensus state: current_term: 1 leader_uuid: "b6af9beb259f4827be6be10aba4a8be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6af9beb259f4827be6be10aba4a8be1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39639 } } }
I20260924 19:30:26.131567 6609 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:26.131491 6610 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6af9beb259f4827be6be10aba4a8be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6af9beb259f4827be6be10aba4a8be1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39639 } } }
I20260924 19:30:26.131731 6610 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:26.131857 6613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:26.132414 6613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:26.133988 6613 catalog_manager.cc:1347] Generated new cluster ID: 9cc941bc3fbd4062b071df21cd651689
I20260924 19:30:26.134045 6613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:26.134320 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.127916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.18.229.190:39639)
0924 19:30:26.128153 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.128161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.128194 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:30:26.128939 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.128947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.129004 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.129144 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:26.129364 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.130761 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.131930 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.132633 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.132735 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.133603 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:26.133607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:26.133608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:26.133806 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:26.133846 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:26.133859 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:26.133899 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.134036 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.134116 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260924 19:30:26.156769 6613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:26.157508 6613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:26.167261 6613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1: Generated new TSK 0
I20260924 19:30:26.167492 6613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:26.169098 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.129:0
--local_ip_for_outbound_sockets=127.18.229.129
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39639
--builtin_ntp_servers=127.18.229.148:32863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:26.250509 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:26.250761 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:26.250799 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:26.252015 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:26.252074 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:26.252117 6628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.129
I20260924 19:30:26.253522 6628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.229.129
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.129
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:26.253803 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:26.254117 6628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:26.259804 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:26.259815 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.260095 6628 server_base.cc:1034] running on GCE node
W20260924 19:30:26.259815 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.260358 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:26.260612 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:26.261767 6628 hybrid_clock.cc:648] HybridClock initialized: now 1790278226261750 us; error 29 us; skew 500 ppm
I20260924 19:30:26.262943 6628 webserver.cc:460] Webserver started at http://127.18.229.129:35115/ using document root <none> and password file <none>
I20260924 19:30:26.263201 6628 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:26.263266 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:26.263401 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:26.264540 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data/instance:
uuid: "1df431c5b86340c99c6adb5d2277c3d1"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "a89664dbb97bf15d08204e92950ce44d"
server_key_iv: "ec5523520e61638e545f4a3c0496930e"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.264891 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance:
uuid: "1df431c5b86340c99c6adb5d2277c3d1"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "a89664dbb97bf15d08204e92950ce44d"
server_key_iv: "ec5523520e61638e545f4a3c0496930e"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.266168 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:30:26.266855 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.267015 6628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:26.267109 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal
uuid: "1df431c5b86340c99c6adb5d2277c3d1"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "a89664dbb97bf15d08204e92950ce44d"
server_key_iv: "ec5523520e61638e545f4a3c0496930e"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.267186 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:26.278558 6628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:26.278868 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:26.279016 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:26.279235 6628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:26.279585 6628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:26.279637 6628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.279672 6628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:26.279698 6628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.285308 6628 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.129:33969
I20260924 19:30:26.285379 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.129:33969 every 8 connection(s)
I20260924 19:30:26.285701 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/data/info.pb
I20260924 19:30:26.290776 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.286485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.129:35607 (local address 127.18.229.190:39639)
0924 19:30:26.286542 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.286544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.286741 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:30:26.287224 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.287225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.287239 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.287286 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:26.287945 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.288668 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.289604 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.289794 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.289842 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.290570 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:26.290571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:26.290571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:26.290575 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:26.290587 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:26.290590 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:26.290635 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.290699 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.290718 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:30:26.290992 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.286468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:39639 (local address 127.18.229.129:35607)
0924 19:30:26.286639 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.286672 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.287143 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.287367 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.287370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.287378 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:26.287870 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.287874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.288746 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.288747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.289531 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.289533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.289863 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.289864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.289968 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.290446 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:26.290447 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:26.290474 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:26.290496 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:26.290637 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:26.290639 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:26.290642 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.290690 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.290763 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260924 19:30:26.291464 6756 heartbeater.cc:347] Connected to a master server at 127.18.229.190:39639
I20260924 19:30:26.291566 6756 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:26.291752 6756 heartbeater.cc:511] Master 127.18.229.190:39639 requested a full tablet report, sending...
I20260924 19:30:26.292258 6568 ts_manager.cc:194] Registered new tserver with Master: 1df431c5b86340c99c6adb5d2277c3d1 (127.18.229.129:33969)
I20260924 19:30:26.292870 6568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.129:35607
I20260924 19:30:26.294426 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6628
I20260924 19:30:26.294541 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-0/wal/instance
I20260924 19:30:26.294730 19350 external_mini_cluster.cc:1428] Setting key 82bc4ef19351db77220a64b8bf26ce67
I20260924 19:30:26.295578 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.130:0
--local_ip_for_outbound_sockets=127.18.229.130
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39639
--builtin_ntp_servers=127.18.229.148:32863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:30:26.374643 6759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:26.374836 6759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:26.374858 6759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:26.375905 6759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:26.375933 6759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:26.375957 6759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.130
I20260924 19:30:26.377318 6759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.229.130
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.130
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:26.377528 6759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:26.377753 6759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:26.382570 6765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:26.382557 6767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:26.382557 6764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.382751 6759 server_base.cc:1034] running on GCE node
I20260924 19:30:26.383056 6759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:26.383342 6759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:26.384497 6759 hybrid_clock.cc:648] HybridClock initialized: now 1790278226384488 us; error 32 us; skew 500 ppm
I20260924 19:30:26.385799 6759 webserver.cc:460] Webserver started at http://127.18.229.130:33263/ using document root <none> and password file <none>
I20260924 19:30:26.386049 6759 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:26.386116 6759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:26.386261 6759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:26.387274 6759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data/instance:
uuid: "bc2997f414f34d228046df6637df8202"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "a74d55b78fda4fa63e9cd9886f0948a7"
server_key_iv: "a8ef02c6e092ab5d18bf9332d8888ff6"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.387611 6759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance:
uuid: "bc2997f414f34d228046df6637df8202"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "a74d55b78fda4fa63e9cd9886f0948a7"
server_key_iv: "a8ef02c6e092ab5d18bf9332d8888ff6"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.388922 6759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:26.389593 6773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.389746 6759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:26.389842 6759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal
uuid: "bc2997f414f34d228046df6637df8202"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "a74d55b78fda4fa63e9cd9886f0948a7"
server_key_iv: "a8ef02c6e092ab5d18bf9332d8888ff6"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.389911 6759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:26.412566 6759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:26.412880 6759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:26.413033 6759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:26.413249 6759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:26.413564 6759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:26.413614 6759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.413651 6759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:26.413679 6759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.419194 6759 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.130:41963
I20260924 19:30:26.419239 6886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.130:41963 every 8 connection(s)
I20260924 19:30:26.419562 6759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/data/info.pb
I20260924 19:30:26.420466 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6759
I20260924 19:30:26.420580 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-1/wal/instance
I20260924 19:30:26.420778 19350 external_mini_cluster.cc:1428] Setting key 8d677f9da5f0658c14b6f3a24523628d
I20260924 19:30:26.421695 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.229.131:0
--local_ip_for_outbound_sockets=127.18.229.131
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39639
--builtin_ntp_servers=127.18.229.148:32863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:30:26.425522 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.420235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.130:37945 (local address 127.18.229.190:39639)
0924 19:30:26.420277 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.420279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.420601 (+ 322us) server_negotiation.cc:407] Connection header received
0924 19:30:26.421229 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.421230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.421241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.421289 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:26.422071 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.422566 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.424051 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.424331 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.424405 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.425271 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:26.425272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:26.425272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:26.425277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:26.425292 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:26.425296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:26.425338 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.425422 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.425447 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:30:26.425741 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.420246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:39639 (local address 127.18.229.130:37945)
0924 19:30:26.420467 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.420501 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.421126 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.421365 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.421369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.421380 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:26.421987 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.421992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.422672 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.422673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.423970 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.423973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.424413 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.424415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.424556 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.425119 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:26.425120 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:26.425148 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:26.425172 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:26.425348 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:26.425353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:26.425356 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.425424 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.425523 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260924 19:30:26.426247 6887 heartbeater.cc:347] Connected to a master server at 127.18.229.190:39639
I20260924 19:30:26.426350 6887 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:26.426520 6887 heartbeater.cc:511] Master 127.18.229.190:39639 requested a full tablet report, sending...
I20260924 19:30:26.426872 6568 ts_manager.cc:194] Registered new tserver with Master: bc2997f414f34d228046df6637df8202 (127.18.229.130:41963)
I20260924 19:30:26.427414 6568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.130:37945
W20260924 19:30:26.502985 6890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:26.503173 6890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:26.503194 6890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:26.504278 6890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:26.504310 6890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:26.504335 6890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.229.131
I20260924 19:30:26.505565 6890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:32863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.229.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.229.131
--webserver_port=0
--tserver_master_addrs=127.18.229.190:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.229.131
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:26.505791 6890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:26.506075 6890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:26.510877 6895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.510983 6890 server_base.cc:1034] running on GCE node
W20260924 19:30:26.510885 6896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:26.510886 6898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:26.511320 6890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:26.511554 6890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:26.512720 6890 hybrid_clock.cc:648] HybridClock initialized: now 1790278226512717 us; error 44 us; skew 500 ppm
I20260924 19:30:26.513945 6890 webserver.cc:460] Webserver started at http://127.18.229.131:38105/ using document root <none> and password file <none>
I20260924 19:30:26.514194 6890 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:26.514259 6890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:26.514415 6890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:26.515447 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data/instance:
uuid: "87949ce5d78949e89b1d40be5f9255ad"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "5b3e6e446eda95934dd43123be00cde4"
server_key_iv: "50f75717761fe394127e8493058b3a16"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.515796 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance:
uuid: "87949ce5d78949e89b1d40be5f9255ad"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "5b3e6e446eda95934dd43123be00cde4"
server_key_iv: "50f75717761fe394127e8493058b3a16"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.517105 6890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:26.517762 6904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.517915 6890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:26.518005 6890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal
uuid: "87949ce5d78949e89b1d40be5f9255ad"
format_stamp: "Formatted at 2026-09-24 19:30:26 on dist-test-slave-r8cc"
server_key: "5b3e6e446eda95934dd43123be00cde4"
server_key_iv: "50f75717761fe394127e8493058b3a16"
server_key_version: "encryptionkey@0"
I20260924 19:30:26.518079 6890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:26.535800 6890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:26.536095 6890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:26.536336 6890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:26.536568 6890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:30:26.536880 6890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:30:26.536927 6890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.536966 6890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:30:26.536998 6890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:26.542481 6890 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.131:43187
I20260924 19:30:26.542555 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.131:43187 every 8 connection(s)
I20260924 19:30:26.542870 6890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/data/info.pb
I20260924 19:30:26.546595 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 6890
I20260924 19:30:26.546737 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal/instance
I20260924 19:30:26.546988 19350 external_mini_cluster.cc:1428] Setting key 7114446e44f0bfb967fe1b09942ae7ce
I20260924 19:30:26.548142 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.543566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:56873 (local address 127.18.229.190:39639)
0924 19:30:26.543619 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.543621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.543860 (+ 239us) server_negotiation.cc:407] Connection header received
0924 19:30:26.544387 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.544388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.544398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.544455 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:26.545064 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.545535 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.546513 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.546755 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.546815 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.547863 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:26.547865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:26.547865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:26.547870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:26.547886 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:26.547890 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:26.547962 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.548033 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.548053 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:30:26.548411 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.543558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:39639 (local address 127.18.229.131:56873)
0924 19:30:26.543726 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.543759 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.544302 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.544510 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.544513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.544521 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:26.545002 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.545006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.545604 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.545604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.546413 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.546415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.546850 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.546852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.547050 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.547705 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:30:26.547706 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:26.547742 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:30:26.547771 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:26.547966 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:26.547970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:26.547973 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.548040 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.548165 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260924 19:30:26.548882 7018 heartbeater.cc:347] Connected to a master server at 127.18.229.190:39639
I20260924 19:30:26.548954 7018 heartbeater.cc:464] Registering TS with master...
I20260924 19:30:26.549136 7018 heartbeater.cc:511] Master 127.18.229.190:39639 requested a full tablet report, sending...
I20260924 19:30:26.549474 6568 ts_manager.cc:194] Registered new tserver with Master: 87949ce5d78949e89b1d40be5f9255ad (127.18.229.131:43187)
I20260924 19:30:26.549800 6568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.229.131:56873
I20260924 19:30:26.558794 19350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:30:26.562608 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.560090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.18.229.190:39639)
0924 19:30:26.560152 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.560153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.560238 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:30:26.560277 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.560278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.560288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.560336 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:26.560450 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.560894 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.561578 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.561748 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.561803 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.562440 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:26.562441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:26.562441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:26.562445 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:26.562456 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:26.562459 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:26.562504 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.562537 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.562555 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:30:26.563928 6568 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:30:26.565430 6568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:30:26.571331 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.567252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41963 (local address 127.0.0.1:50354)
0924 19:30:26.567798 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.567811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.567866 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.568102 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.568103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.568104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:26.568167 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.568169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.569013 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.569014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.570342 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.570345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.571039 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.571040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.571108 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.571121 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.571191 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.571241 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":17,"thread_start_us":97,"threads_started":1}
I20260924 19:30:26.571350 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.567411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.18.229.130:41963)
0924 19:30:26.567598 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.567603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.567866 (+ 263us) server_negotiation.cc:407] Connection header received
0924 19:30:26.567933 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.567935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.567950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.568033 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:26.568245 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.568922 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.570437 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.570949 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.571039 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.571124 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.571207 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.571241 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260924 19:30:26.571624 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.567425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.18.229.129:33969)
0924 19:30:26.567684 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.567688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.567698 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:26.567772 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.567774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.567790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.567872 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:26.568114 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.569223 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.570443 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.571106 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.571179 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.571255 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.571479 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.571539 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260924 19:30:26.571666 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.567246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:33969 (local address 127.0.0.1:40458)
0924 19:30:26.567594 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.567631 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.567696 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.567947 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.567950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.567953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:26.568030 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.568032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.569310 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.569311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.570383 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.570385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.571308 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.571309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.571371 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.571375 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.571426 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.571554 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":55,"thread_start_us":63,"threads_started":1}
I20260924 19:30:26.571734 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.567456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.18.229.131:43187)
0924 19:30:26.567649 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.567652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.567686 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:30:26.567778 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.567780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.567793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.567872 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:26.568113 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.569264 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.570391 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.571122 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.571179 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.571256 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.571592 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.571647 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260924 19:30:26.571800 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.567248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.131:43187 (local address 127.0.0.1:38136)
0924 19:30:26.567595 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.567631 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.567696 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.567953 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.567954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.567955 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:26.568030 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.568033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.569362 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.569362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.570306 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.570308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.571313 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.571314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.571371 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.571375 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.571425 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.571502 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":111,"threads_started":1}
I20260924 19:30:26.572412 6952 tablet_service.cc:1467] Processing CreateTablet for tablet 8b931f12fa5d45c984e33eaa0c824929 (DEFAULT_TABLE table=test-table [id=834a76bb04a34ef2b949e3c3c7b99431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:26.572461 6690 tablet_service.cc:1467] Processing CreateTablet for tablet 8b931f12fa5d45c984e33eaa0c824929 (DEFAULT_TABLE table=test-table [id=834a76bb04a34ef2b949e3c3c7b99431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:26.572760 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b931f12fa5d45c984e33eaa0c824929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:26.572785 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b931f12fa5d45c984e33eaa0c824929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:26.572863 6821 tablet_service.cc:1467] Processing CreateTablet for tablet 8b931f12fa5d45c984e33eaa0c824929 (DEFAULT_TABLE table=test-table [id=834a76bb04a34ef2b949e3c3c7b99431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:30:26.573159 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b931f12fa5d45c984e33eaa0c824929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:26.575254 7032 tablet_bootstrap.cc:492] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: Bootstrap starting.
I20260924 19:30:26.575881 7032 tablet_bootstrap.cc:654] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:26.576269 7033 tablet_bootstrap.cc:492] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: Bootstrap starting.
I20260924 19:30:26.576371 7032 log.cc:824] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:26.576490 7034 tablet_bootstrap.cc:492] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Bootstrap starting.
I20260924 19:30:26.577009 7033 tablet_bootstrap.cc:654] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:26.577092 7034 tablet_bootstrap.cc:654] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:26.577198 7032 tablet_bootstrap.cc:492] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: No bootstrap required, opened a new log
I20260924 19:30:26.577272 7032 ts_tablet_manager.cc:1397] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:30:26.577337 7033 log.cc:824] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:26.577412 7034 log.cc:824] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:26.578111 7033 tablet_bootstrap.cc:492] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: No bootstrap required, opened a new log
I20260924 19:30:26.578203 7033 ts_tablet_manager.cc:1397] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:30:26.579977 7032 raft_consensus.cc:348] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.580185 7032 raft_consensus.cc:374] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:26.580235 7032 raft_consensus.cc:729] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df431c5b86340c99c6adb5d2277c3d1, State: Initialized, Role: FOLLOWER
I20260924 19:30:26.580349 7032 consensus_queue.cc:260] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.580610 7032 ts_tablet_manager.cc:1428] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 19:30:26.580641 6756 heartbeater.cc:503] Master 127.18.229.190:39639 was elected leader, sending a full tablet report...
I20260924 19:30:26.580791 7033 raft_consensus.cc:348] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.580951 7033 raft_consensus.cc:374] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:26.581017 7033 raft_consensus.cc:729] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2997f414f34d228046df6637df8202, State: Initialized, Role: FOLLOWER
I20260924 19:30:26.581137 7033 consensus_queue.cc:260] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.581384 7033 ts_tablet_manager.cc:1428] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:30:26.581418 6887 heartbeater.cc:503] Master 127.18.229.190:39639 was elected leader, sending a full tablet report...
I20260924 19:30:26.582199 7034 tablet_bootstrap.cc:492] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: No bootstrap required, opened a new log
I20260924 19:30:26.582280 7034 ts_tablet_manager.cc:1397] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 19:30:26.584003 7034 raft_consensus.cc:348] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.584152 7034 raft_consensus.cc:374] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:26.584193 7034 raft_consensus.cc:729] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87949ce5d78949e89b1d40be5f9255ad, State: Initialized, Role: FOLLOWER
I20260924 19:30:26.584287 7034 consensus_queue.cc:260] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.584461 7034 ts_tablet_manager.cc:1428] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:30:26.584534 7018 heartbeater.cc:503] Master 127.18.229.190:39639 was elected leader, sending a full tablet report...
I20260924 19:30:26.611624 7040 raft_consensus.cc:482] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:30:26.611748 7040 raft_consensus.cc:504] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.612088 7040 leader_election.cc:288] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df431c5b86340c99c6adb5d2277c3d1 (127.18.229.129:33969), bc2997f414f34d228046df6637df8202 (127.18.229.130:41963)
I20260924 19:30:26.615871 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.612335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:51929 (local address 127.18.229.130:41963)
0924 19:30:26.612380 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.612382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.612605 (+ 223us) server_negotiation.cc:407] Connection header received
0924 19:30:26.612638 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.612638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.612646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.612699 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:26.612823 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.613523 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.614983 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.615542 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.615607 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.615615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.615761 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.615802 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:30:26.615871 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.612369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.229.131:51137 (local address 127.18.229.129:33969)
0924 19:30:26.612416 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.612418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.612426 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:26.612456 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.612456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.612470 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.612549 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:30:26.612727 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.613525 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.614983 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.615527 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.615607 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.615615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.615761 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.615805 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:30:26.615886 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.612284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:33969 (local address 127.18.229.131:51137)
0924 19:30:26.612345 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.612357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.612399 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.612559 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.612560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.612562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:26.612647 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.612650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.613605 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.613605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.614894 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.614897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.615614 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.615615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.615687 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.615690 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.615747 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.615796 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":67}
I20260924 19:30:26.615906 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.612276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41963 (local address 127.18.229.131:51929)
0924 19:30:26.612472 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.612482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:26.612544 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:26.612690 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:26.612691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:26.612693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:30:26.612752 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.612753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.613605 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.613605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.614895 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:26.614897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.615627 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.615627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:26.615687 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.615690 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:30:26.615747 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:30:26.615797 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":67,"thread_start_us":116,"threads_started":1}
I20260924 19:30:26.616261 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b931f12fa5d45c984e33eaa0c824929" candidate_uuid: "87949ce5d78949e89b1d40be5f9255ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df431c5b86340c99c6adb5d2277c3d1" is_pre_election: true
I20260924 19:30:26.616415 6710 raft_consensus.cc:2454] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87949ce5d78949e89b1d40be5f9255ad in term 0.
I20260924 19:30:26.616470 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b931f12fa5d45c984e33eaa0c824929" candidate_uuid: "87949ce5d78949e89b1d40be5f9255ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2997f414f34d228046df6637df8202" is_pre_election: true
I20260924 19:30:26.616623 6841 raft_consensus.cc:2454] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87949ce5d78949e89b1d40be5f9255ad in term 0.
I20260924 19:30:26.616626 6906 leader_election.cc:302] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df431c5b86340c99c6adb5d2277c3d1, 87949ce5d78949e89b1d40be5f9255ad; no voters:
I20260924 19:30:26.616731 7040 raft_consensus.cc:2781] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:30:26.616801 7040 raft_consensus.cc:482] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:30:26.616832 7040 raft_consensus.cc:3037] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:26.617523 7040 raft_consensus.cc:504] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.617676 7040 leader_election.cc:288] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [CANDIDATE]: Term 1 election: Requested vote from peers 1df431c5b86340c99c6adb5d2277c3d1 (127.18.229.129:33969), bc2997f414f34d228046df6637df8202 (127.18.229.130:41963)
I20260924 19:30:26.617820 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b931f12fa5d45c984e33eaa0c824929" candidate_uuid: "87949ce5d78949e89b1d40be5f9255ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df431c5b86340c99c6adb5d2277c3d1"
I20260924 19:30:26.617856 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b931f12fa5d45c984e33eaa0c824929" candidate_uuid: "87949ce5d78949e89b1d40be5f9255ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2997f414f34d228046df6637df8202"
I20260924 19:30:26.617905 6710 raft_consensus.cc:3037] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:26.617939 6841 raft_consensus.cc:3037] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:26.618763 6710 raft_consensus.cc:2454] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87949ce5d78949e89b1d40be5f9255ad in term 1.
I20260924 19:30:26.618767 6841 raft_consensus.cc:2454] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87949ce5d78949e89b1d40be5f9255ad in term 1.
I20260924 19:30:26.618927 6908 leader_election.cc:302] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87949ce5d78949e89b1d40be5f9255ad, bc2997f414f34d228046df6637df8202; no voters:
I20260924 19:30:26.619099 7040 raft_consensus.cc:2781] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:26.619316 7040 raft_consensus.cc:686] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 1 LEADER]: Becoming Leader. State: Replica: 87949ce5d78949e89b1d40be5f9255ad, State: Running, Role: LEADER
I20260924 19:30:26.619414 7040 consensus_queue.cc:237] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } }
I20260924 19:30:26.620220 6567 catalog_manager.cc:5495] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 87949ce5d78949e89b1d40be5f9255ad (127.18.229.131). New cstate: current_term: 1 leader_uuid: "87949ce5d78949e89b1d40be5f9255ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87949ce5d78949e89b1d40be5f9255ad" member_type: VOTER last_known_addr { host: "127.18.229.131" port: 43187 } health_report { overall_health: HEALTHY } } }
I20260924 19:30:26.635398 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.633329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.18.229.131:43187)
0924 19:30:26.633384 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:26.633385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:26.633411 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:30:26.633443 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:26.633444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:26.633461 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:26.633506 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:26.633603 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.634049 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.634769 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:26.634939 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:26.634985 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:26.635100 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:26.635191 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:26.635245 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:26.635258 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 19:30:26.635278 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:30:26.637091 6841 raft_consensus.cc:1261] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 1 FOLLOWER]: Refusing update from remote peer 87949ce5d78949e89b1d40be5f9255ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:26.637115 6710 raft_consensus.cc:1261] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 1 FOLLOWER]: Refusing update from remote peer 87949ce5d78949e89b1d40be5f9255ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:26.637403 7040 consensus_queue.cc:1035] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2997f414f34d228046df6637df8202" member_type: VOTER last_known_addr { host: "127.18.229.130" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:26.637539 7042 consensus_queue.cc:1035] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df431c5b86340c99c6adb5d2277c3d1" member_type: VOTER last_known_addr { host: "127.18.229.129" port: 33969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:26.639809 7045 mvcc.cc:205] Tried to move back new op lower bound from 7332979616303194112 to 7332979616233795584. Current Snapshot: MvccSnapshot[applied={T|T < 7332979616303194112}]
I20260924 19:30:26.640172 7046 mvcc.cc:205] Tried to move back new op lower bound from 7332979616303194112 to 7332979616233795584. Current Snapshot: MvccSnapshot[applied={T|T < 7332979616303194112}]
I20260924 19:30:26.641947 6567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:26.642023 6567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:30:26.642740 6567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6af9beb259f4827be6be10aba4a8be1: Sending DeleteTablet for 3 replicas of tablet 8b931f12fa5d45c984e33eaa0c824929
I20260924 19:30:26.643049 6821 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b931f12fa5d45c984e33eaa0c824929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:26 UTC) from {username='slave'} at 127.0.0.1:50354
I20260924 19:30:26.643062 6690 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b931f12fa5d45c984e33eaa0c824929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:26 UTC) from {username='slave'} at 127.0.0.1:40458
I20260924 19:30:26.643249 6952 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b931f12fa5d45c984e33eaa0c824929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:30:26 UTC) from {username='slave'} at 127.0.0.1:38136
I20260924 19:30:26.643285 7055 tablet_replica.cc:330] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1: stopping tablet replica
I20260924 19:30:26.643381 7055 raft_consensus.cc:2229] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:26.643472 7055 raft_consensus.cc:2258] T 8b931f12fa5d45c984e33eaa0c824929 P 1df431c5b86340c99c6adb5d2277c3d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:26.643481 7057 tablet_replica.cc:330] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: stopping tablet replica
I20260924 19:30:26.643623 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6628
I20260924 19:30:26.643658 7057 raft_consensus.cc:2229] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:30:26.644424 7057 raft_consensus.cc:2258] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:26.644824 7057 ts_tablet_manager.cc:1905] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:30:26.646775 7057 ts_tablet_manager.cc:1918] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:30:26.646880 7057 log.cc:1196] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Deleting WAL directory at /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790277987605586-19350-0/minicluster-data/ts-2/wal/wals/8b931f12fa5d45c984e33eaa0c824929
I20260924 19:30:26.647186 7057 ts_tablet_manager.cc:1939] T 8b931f12fa5d45c984e33eaa0c824929 P 87949ce5d78949e89b1d40be5f9255ad: Deleting consensus metadata
I20260924 19:30:26.647629 6552 catalog_manager.cc:4841] TS 87949ce5d78949e89b1d40be5f9255ad (127.18.229.131:43187): tablet 8b931f12fa5d45c984e33eaa0c824929 (table test-table [id=834a76bb04a34ef2b949e3c3c7b99431]) successfully deleted
W20260924 19:30:26.650012 6553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.229.129:33969 (error 108)
I20260924 19:30:26.650182 7056 tablet_replica.cc:330] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202: stopping tablet replica
I20260924 19:30:26.650310 7056 raft_consensus.cc:2229] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:30:26.650463 7056 raft_consensus.cc:2258] T 8b931f12fa5d45c984e33eaa0c824929 P bc2997f414f34d228046df6637df8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:30:26.650503 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.650385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.129:33969 (local address 127.0.0.1:40474)
0924 19:30:26.650433 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.650478 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.129:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:30:26.650635 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6759
W20260924 19:30:26.650631 6553 catalog_manager.cc:4584] TS 1df431c5b86340c99c6adb5d2277c3d1 (127.18.229.129:33969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b931f12fa5d45c984e33eaa0c824929: Network error: Client connection negotiation failed: client connection to 127.18.229.129:33969: connect: Connection refused (error 111)
I20260924 19:30:26.655895 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:26.655787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.130:41963 (local address 127.0.0.1:50370)
0924 19:30:26.655837 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:26.655865 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.130:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:30:26.655910 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6890
I20260924 19:30:26.660506 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 6537
2026-09-24T19:30:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (673 ms)
[----------] 6 tests from EncryptionPolicyTest (8582 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:26 dist-test-slave-r8cc krb5kdc[7061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:26 dist-test-slave-r8cc krb5kdc[7061](info): set up 2 sockets
Sep 24 19:30:26 dist-test-slave-r8cc krb5kdc[7061](info): commencing operation
krb5kdc: starting...
W20260924 19:30:27.683976 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:27 dist-test-slave-r8cc krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:27.796865 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:27.879022 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:27.879242 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:27.879288 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:27.880497 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:27.880551 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:27.880575 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:27.880604 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:27.880633 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:27.882061 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:27.882328 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:27.882539 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:27.887246 7079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:27.887218 7082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:27.887233 7080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:27.887292 7074 server_base.cc:1034] running on GCE node
I20260924 19:30:27.887651 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:27.887961 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:27.889112 7074 hybrid_clock.cc:648] HybridClock initialized: now 1790278227889084 us; error 55 us; skew 500 ppm
Sep 24 19:30:27 dist-test-slave-r8cc krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:27.891449 7074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:27.891909 7074 webserver.cc:460] Webserver started at http://127.0.0.1:38361/ using document root <none> and password file <none>
I20260924 19:30:27.892184 7074 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:27.892246 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:27.892398 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:27.893414 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "9746062878784b7bb423a3b4cf57316d"
format_stamp: "Formatted at 2026-09-24 19:30:27 on dist-test-slave-r8cc"
server_key: "a1ef37f7f3d114dcc4907dd03ce4590e"
server_key_iv: "dc6b2872983d04935d7a27dd01230d82"
server_key_version: "encryptionkey@0"
I20260924 19:30:27.893760 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "9746062878784b7bb423a3b4cf57316d"
format_stamp: "Formatted at 2026-09-24 19:30:27 on dist-test-slave-r8cc"
server_key: "a1ef37f7f3d114dcc4907dd03ce4590e"
server_key_iv: "dc6b2872983d04935d7a27dd01230d82"
server_key_version: "encryptionkey@0"
I20260924 19:30:27.895015 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:27.895660 7089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:27.895833 7074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:27.895929 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "9746062878784b7bb423a3b4cf57316d"
format_stamp: "Formatted at 2026-09-24 19:30:27 on dist-test-slave-r8cc"
server_key: "a1ef37f7f3d114dcc4907dd03ce4590e"
server_key_iv: "dc6b2872983d04935d7a27dd01230d82"
server_key_version: "encryptionkey@0"
I20260924 19:30:27.896003 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:27.908378 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:27.910073 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:27.910221 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:27.914181 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42441
I20260924 19:30:27.914228 7141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42441 every 8 connection(s)
I20260924 19:30:27.914565 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:27.915364 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:27.917428 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d: Bootstrap starting.
I20260924 19:30:27.917898 7142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:27.918118 7142 log.cc:824] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:27.918689 7142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d: No bootstrap required, opened a new log
I20260924 19:30:27.920179 7142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9746062878784b7bb423a3b4cf57316d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42441 } }
I20260924 19:30:27.920331 7142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:27.920353 7142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9746062878784b7bb423a3b4cf57316d, State: Initialized, Role: FOLLOWER
I20260924 19:30:27.920452 7142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9746062878784b7bb423a3b4cf57316d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42441 } }
I20260924 19:30:27.920498 7142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:27.920517 7142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:27.920547 7142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:27.920979 7142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9746062878784b7bb423a3b4cf57316d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42441 } }
I20260924 19:30:27.921093 7142 leader_election.cc:302] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9746062878784b7bb423a3b4cf57316d; no voters:
I20260924 19:30:27.921298 7142 leader_election.cc:288] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:27.921350 7145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:27.921377 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7074
I20260924 19:30:27.921470 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:27.921537 7142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:27.921541 7145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [term 1 LEADER]: Becoming Leader. State: Replica: 9746062878784b7bb423a3b4cf57316d, State: Running, Role: LEADER
I20260924 19:30:27.921657 7145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9746062878784b7bb423a3b4cf57316d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42441 } }
I20260924 19:30:27.921715 19350 external_mini_cluster.cc:1428] Setting key 8bc51dddd9fb3ef6eeba57fa16ce7324
I20260924 19:30:27.922019 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9746062878784b7bb423a3b4cf57316d. Latest consensus state: current_term: 1 leader_uuid: "9746062878784b7bb423a3b4cf57316d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9746062878784b7bb423a3b4cf57316d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42441 } } }
I20260924 19:30:27.922150 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:27.922345 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9746062878784b7bb423a3b4cf57316d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9746062878784b7bb423a3b4cf57316d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42441 } } }
I20260924 19:30:27.922438 7154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:27.923048 7154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:27.923717 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:27.925132 7154 catalog_manager.cc:1347] Generated new cluster ID: 27c7484f2df34138ade5a5e105daf4e8
I20260924 19:30:27.925179 7154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:30:27 dist-test-slave-r8cc krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:30:27.935834 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:27.923246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39003 (local address 127.0.0.1:42441)
0924 19:30:27.923888 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:27.923897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:27.923938 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:30:27.924219 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:27.924226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:27.924282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:27.924411 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:27.925433 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:27.926794 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:27.927869 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:27.928529 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:27.928629 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:27.931448 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:27.931452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:27.931452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:27.931464 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:27.933749 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:27.934234 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:27.934237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:27.934237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:27.934357 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:27.934596 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:27.934597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:27.934597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:27.934908 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:27.935057 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:27.935514 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 19:30:27.935634 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":82,"threads_started":1}
I20260924 19:30:27.936780 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:27.941660 7154 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:27.942101 7154 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:27.945062 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:27.938161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51041 (local address 127.0.0.1:42441)
0924 19:30:27.938200 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:27.938202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:27.938356 (+ 154us) server_negotiation.cc:407] Connection header received
0924 19:30:27.938384 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:27.938385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:27.938394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:27.938465 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:27.938975 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:27.939738 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:27.940726 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:27.941033 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:27.941098 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:27.942378 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:27.942379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:27.942380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:27.942386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:27.943848 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:27.944314 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:27.944315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:27.944315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:27.944369 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:27.944589 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:27.944589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:27.944590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:27.944697 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:27.944803 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:27.944926 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:30:27.944981 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:30:27.946312 7154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9746062878784b7bb423a3b4cf57316d: Generated new TSK 0
I20260924 19:30:27.946492 7154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:27.948854 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7074
W20260924 19:30:27.953198 7059 connection.cc:535] client connection to 127.0.0.1:42441 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42441): Connection reset by peer (error 104)
2026-09-24T19:30:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:27 dist-test-slave-r8cc krb5kdc[7173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:27 dist-test-slave-r8cc krb5kdc[7173](info): set up 2 sockets
Sep 24 19:30:27 dist-test-slave-r8cc krb5kdc[7173](info): commencing operation
krb5kdc: starting...
W20260924 19:30:28.976001 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:29 dist-test-slave-r8cc krb5kdc[7173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:29.091245 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:29.173040 7186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:29.173250 7186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:29.173274 7186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:29.174435 7186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:29.174479 7186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:29.174494 7186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:29.174507 7186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:29.174561 7186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:29.176134 7186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:29.176384 7186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:29.176607 7186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:29.181798 7194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:29.181785 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:29.182040 7186 server_base.cc:1034] running on GCE node
W20260924 19:30:29.181799 7191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:29.182329 7186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:29.182614 7186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:29.183765 7186 hybrid_clock.cc:648] HybridClock initialized: now 1790278229183757 us; error 33 us; skew 500 ppm
Sep 24 19:30:29 dist-test-slave-r8cc krb5kdc[7173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:29.191236 7186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:29.191727 7186 webserver.cc:460] Webserver started at http://127.0.0.1:33243/ using document root <none> and password file <none>
I20260924 19:30:29.191967 7186 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:29.192030 7186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:29.192206 7186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:29.193279 7186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae"
format_stamp: "Formatted at 2026-09-24 19:30:29 on dist-test-slave-r8cc"
server_key: "68a7f25b5d64498e9b1859b6086d9a1b"
server_key_iv: "000315303de1f8de49c229e122656911"
server_key_version: "encryptionkey@0"
I20260924 19:30:29.193645 7186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae"
format_stamp: "Formatted at 2026-09-24 19:30:29 on dist-test-slave-r8cc"
server_key: "68a7f25b5d64498e9b1859b6086d9a1b"
server_key_iv: "000315303de1f8de49c229e122656911"
server_key_version: "encryptionkey@0"
I20260924 19:30:29.194967 7186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:30:29.195684 7201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:29.195843 7186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:29.195937 7186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae"
format_stamp: "Formatted at 2026-09-24 19:30:29 on dist-test-slave-r8cc"
server_key: "68a7f25b5d64498e9b1859b6086d9a1b"
server_key_iv: "000315303de1f8de49c229e122656911"
server_key_version: "encryptionkey@0"
I20260924 19:30:29.196012 7186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:29.217770 7186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:29.219586 7186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:29.219743 7186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:29.223717 7186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43327
I20260924 19:30:29.223754 7253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43327 every 8 connection(s)
I20260924 19:30:29.224205 7186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:29.225057 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:29.225945 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7186
I20260924 19:30:29.226037 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:29.226239 19350 external_mini_cluster.cc:1428] Setting key 428dd871774e63a4b132739c2247b031
I20260924 19:30:29.227979 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae: Bootstrap starting.
I20260924 19:30:29.228672 7254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:29.228986 7254 log.cc:824] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:29.229677 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae: No bootstrap required, opened a new log
I20260924 19:30:29.232071 7254 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260924 19:30:29.232265 7254 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:29.232312 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:29.226755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48989 (local address 127.0.0.1:43327)
0924 19:30:29.226951 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:29.226960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:29.226992 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:30:29.227676 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:29.227685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:29.227747 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:29.227892 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:29.228074 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:29.229435 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:29.230203 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:29.230821 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:29.230936 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:29.231599 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:29.231604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:29.231604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:29.231847 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:29.231884 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:29.231899 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:29.231980 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:29.232059 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:30:29.232087 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260924 19:30:29.232301 7254 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd152e3c1a5c4a9d98ab6521ae2b11ae, State: Initialized, Role: FOLLOWER
I20260924 19:30:29.232473 7254 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260924 19:30:29.232561 7254 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:29.232594 7254 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:29.232641 7254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:29.233269 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:29.233341 7254 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260924 19:30:29.233448 7254 leader_election.cc:302] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd152e3c1a5c4a9d98ab6521ae2b11ae; no voters:
I20260924 19:30:29.233613 7254 leader_election.cc:288] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:29.233652 7259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:29.233814 7259 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [term 1 LEADER]: Becoming Leader. State: Replica: bd152e3c1a5c4a9d98ab6521ae2b11ae, State: Running, Role: LEADER
I20260924 19:30:29.233876 7254 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:29.233912 7259 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260924 19:30:29.234223 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd152e3c1a5c4a9d98ab6521ae2b11ae. Latest consensus state: current_term: 1 leader_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } } }
I20260924 19:30:29.234309 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:29.234237 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd152e3c1a5c4a9d98ab6521ae2b11ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } } }
I20260924 19:30:29.234478 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae [sys.catalog]: This master's current role is: LEADER
W20260924 19:30:29.235548 7279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:29.235597 7279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:29.235730 7268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:29.235919 7268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:29.237498 7268 catalog_manager.cc:1347] Generated new cluster ID: 1c3fef98ae394e84bbfd0b9176a8bc30
I20260924 19:30:29.237548 7268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:29.238456 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:29.234856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48097 (local address 127.0.0.1:43327)
0924 19:30:29.234888 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:29.234890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:29.234994 (+ 104us) server_negotiation.cc:407] Connection header received
0924 19:30:29.235057 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:29.235058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:29.235070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:29.235133 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:29.235367 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:29.236056 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:29.237047 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:29.237344 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:29.237415 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:29.238191 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:29.238192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:29.238192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:29.238197 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:29.238213 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:29.238217 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:29.238267 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:29.238377 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:30:29.238387 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 19:30:29.249161 7268 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:29.249693 7268 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:29.259255 7268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd152e3c1a5c4a9d98ab6521ae2b11ae: Generated new TSK 0
I20260924 19:30:29.259449 7268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:29.262811 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7186
W20260924 19:30:29.266762 7171 connection.cc:535] client connection to 127.0.0.1:43327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43327): Connection reset by peer (error 104)
2026-09-24T19:30:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:29 dist-test-slave-r8cc krb5kdc[7285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:29 dist-test-slave-r8cc krb5kdc[7285](info): set up 2 sockets
Sep 24 19:30:29 dist-test-slave-r8cc krb5kdc[7285](info): commencing operation
krb5kdc: starting...
W20260924 19:30:30.292069 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:30 dist-test-slave-r8cc krb5kdc[7285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:30.404811 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:30.485466 7298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:30.485692 7298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:30.485716 7298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:30.486855 7298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:30.486894 7298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:30.486908 7298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:30.486922 7298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:30.486933 7298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:30.488375 7298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:30.488572 7298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:30.488754 7298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:30.493570 7304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:30.493554 7306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:30.493539 7303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:30.493628 7298 server_base.cc:1034] running on GCE node
I20260924 19:30:30.493952 7298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:30.494246 7298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:30.495402 7298 hybrid_clock.cc:648] HybridClock initialized: now 1790278230495376 us; error 43 us; skew 500 ppm
Sep 24 19:30:30 dist-test-slave-r8cc krb5kdc[7285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:30.497793 7298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:30.498179 7298 webserver.cc:460] Webserver started at http://127.0.0.1:36383/ using document root <none> and password file <none>
I20260924 19:30:30.498401 7298 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:30.498481 7298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:30.498616 7298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:30.499701 7298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "edd3e7b8128542b1aa0aea9e5343e177"
format_stamp: "Formatted at 2026-09-24 19:30:30 on dist-test-slave-r8cc"
server_key: "ece0e18de31287ac8083f89010d40e7b"
server_key_iv: "1426425c50a9ac87c4b2f4847731c575"
server_key_version: "encryptionkey@0"
I20260924 19:30:30.500075 7298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "edd3e7b8128542b1aa0aea9e5343e177"
format_stamp: "Formatted at 2026-09-24 19:30:30 on dist-test-slave-r8cc"
server_key: "ece0e18de31287ac8083f89010d40e7b"
server_key_iv: "1426425c50a9ac87c4b2f4847731c575"
server_key_version: "encryptionkey@0"
I20260924 19:30:30.501365 7298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:30.502120 7313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:30.502318 7298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:30.502416 7298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "edd3e7b8128542b1aa0aea9e5343e177"
format_stamp: "Formatted at 2026-09-24 19:30:30 on dist-test-slave-r8cc"
server_key: "ece0e18de31287ac8083f89010d40e7b"
server_key_iv: "1426425c50a9ac87c4b2f4847731c575"
server_key_version: "encryptionkey@0"
I20260924 19:30:30.502486 7298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:30.505726 7298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:30.507478 7298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:30.507622 7298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:30.511353 7298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40929
I20260924 19:30:30.511394 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40929 every 8 connection(s)
I20260924 19:30:30.511734 7298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:30.512701 7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:30.514969 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177: Bootstrap starting.
I20260924 19:30:30.515533 7366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:30.515811 7366 log.cc:824] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:30.516571 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177: No bootstrap required, opened a new log
I20260924 19:30:30.518162 7366 raft_consensus.cc:348] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd3e7b8128542b1aa0aea9e5343e177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40929 } }
I20260924 19:30:30.518301 7366 raft_consensus.cc:374] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:30.518368 7366 raft_consensus.cc:729] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd3e7b8128542b1aa0aea9e5343e177, State: Initialized, Role: FOLLOWER
I20260924 19:30:30.518543 7366 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd3e7b8128542b1aa0aea9e5343e177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40929 } }
I20260924 19:30:30.518606 7366 raft_consensus.cc:388] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:30.518642 7366 raft_consensus.cc:482] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:30.518689 7366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:30.519227 7366 raft_consensus.cc:504] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd3e7b8128542b1aa0aea9e5343e177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40929 } }
I20260924 19:30:30.519336 7366 leader_election.cc:302] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edd3e7b8128542b1aa0aea9e5343e177; no voters:
I20260924 19:30:30.519377 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7298
I20260924 19:30:30.519501 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:30.519516 7366 leader_election.cc:288] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:30.519537 7369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:30.519659 7369 raft_consensus.cc:686] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [term 1 LEADER]: Becoming Leader. State: Replica: edd3e7b8128542b1aa0aea9e5343e177, State: Running, Role: LEADER
I20260924 19:30:30.519742 19350 external_mini_cluster.cc:1428] Setting key c6cacba7c938ad86aaa9d2ba3afe2451
I20260924 19:30:30.519738 7369 consensus_queue.cc:237] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd3e7b8128542b1aa0aea9e5343e177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40929 } }
I20260924 19:30:30.519780 7366 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:30.520042 7370 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edd3e7b8128542b1aa0aea9e5343e177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd3e7b8128542b1aa0aea9e5343e177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40929 } } }
I20260924 19:30:30.520176 7370 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:30.520423 7371 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edd3e7b8128542b1aa0aea9e5343e177. Latest consensus state: current_term: 1 leader_uuid: "edd3e7b8128542b1aa0aea9e5343e177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd3e7b8128542b1aa0aea9e5343e177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40929 } } }
I20260924 19:30:30.520525 7374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:30.520545 7371 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:30.521597 7374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:30.522051 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:30.520237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46671 (local address 127.0.0.1:40929)
0924 19:30:30.520441 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:30.520450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:30.520484 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:30:30.520612 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:30.520621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:30.520659 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:30.520826 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:30.521543 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:30.521547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:30.521547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:30.521552 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:30.521595 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:30.521610 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:30.521691 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:30.521841 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:30:30.521883 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260924 19:30:30.522933 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:30.524346 7374 catalog_manager.cc:1347] Generated new cluster ID: d13acc856f1b4b13ac6072065b57c414
I20260924 19:30:30.524400 7374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:30.525588 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:30.524459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51905 (local address 127.0.0.1:40929)
0924 19:30:30.524493 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:30.524495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:30.524695 (+ 200us) server_negotiation.cc:407] Connection header received
0924 19:30:30.524758 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:30.524759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:30.524765 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:30.524814 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:30.525377 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:30.525378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:30.525379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:30.525383 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:30.525396 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:30.525400 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:30.525454 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:30.525507 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:30:30.525521 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:30:30.542768 7374 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:30.542968 7374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:30.561278 7374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edd3e7b8128542b1aa0aea9e5343e177: Generated new TSK 0
I20260924 19:30:30.561512 7374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:30.569653 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7298
2026-09-24T19:30:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:30 dist-test-slave-r8cc krb5kdc[7397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:30 dist-test-slave-r8cc krb5kdc[7397](info): set up 2 sockets
Sep 24 19:30:30 dist-test-slave-r8cc krb5kdc[7397](info): commencing operation
krb5kdc: starting...
W20260924 19:30:31.592216 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:31 dist-test-slave-r8cc krb5kdc[7397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:31.702965 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:31.782640 7410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:31.782842 7410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:31.782864 7410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:31.784030 7410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:31.784073 7410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:31.784088 7410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:31.784160 7410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:31.784181 7410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:31.785676 7410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:31.785915 7410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:31.786118 7410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:31.790897 7415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:31.790956 7410 server_base.cc:1034] running on GCE node
W20260924 19:30:31.790906 7416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:31.790906 7418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:31.791356 7410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:31.791635 7410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:31.792785 7410 hybrid_clock.cc:648] HybridClock initialized: now 1790278231792775 us; error 39 us; skew 500 ppm
Sep 24 19:30:31 dist-test-slave-r8cc krb5kdc[7397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:31.795091 7410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:31.795501 7410 webserver.cc:460] Webserver started at http://127.0.0.1:34703/ using document root <none> and password file <none>
I20260924 19:30:31.795701 7410 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:31.795738 7410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:31.795825 7410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:31.796840 7410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "edde746cc9de43b0b955f0413ed810b4"
format_stamp: "Formatted at 2026-09-24 19:30:31 on dist-test-slave-r8cc"
server_key: "0010a1e0acd9f3c376e725b8a28f7c5f"
server_key_iv: "974c0fa8e3364b58a156096b27183041"
server_key_version: "encryptionkey@0"
I20260924 19:30:31.797125 7410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "edde746cc9de43b0b955f0413ed810b4"
format_stamp: "Formatted at 2026-09-24 19:30:31 on dist-test-slave-r8cc"
server_key: "0010a1e0acd9f3c376e725b8a28f7c5f"
server_key_iv: "974c0fa8e3364b58a156096b27183041"
server_key_version: "encryptionkey@0"
I20260924 19:30:31.798312 7410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:31.798919 7425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:31.799077 7410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:31.799180 7410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "edde746cc9de43b0b955f0413ed810b4"
format_stamp: "Formatted at 2026-09-24 19:30:31 on dist-test-slave-r8cc"
server_key: "0010a1e0acd9f3c376e725b8a28f7c5f"
server_key_iv: "974c0fa8e3364b58a156096b27183041"
server_key_version: "encryptionkey@0"
I20260924 19:30:31.799248 7410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:31.806995 7410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:31.808722 7410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:31.808866 7410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:31.812569 7410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34407
I20260924 19:30:31.812597 7477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34407 every 8 connection(s)
I20260924 19:30:31.812929 7410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:31.813735 7478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:31.815794 7478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4: Bootstrap starting.
I20260924 19:30:31.816368 7478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:31.816588 7478 log.cc:824] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:31.817180 7478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4: No bootstrap required, opened a new log
I20260924 19:30:31.817610 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7410
I20260924 19:30:31.817719 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:31.817930 19350 external_mini_cluster.cc:1428] Setting key 2a3a8bca86f3d9e95ccd0f9288a55675
I20260924 19:30:31.819114 7478 raft_consensus.cc:348] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde746cc9de43b0b955f0413ed810b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 19:30:31.819284 7478 raft_consensus.cc:374] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:31.819332 7478 raft_consensus.cc:729] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edde746cc9de43b0b955f0413ed810b4, State: Initialized, Role: FOLLOWER
I20260924 19:30:31.819470 7478 consensus_queue.cc:260] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde746cc9de43b0b955f0413ed810b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 19:30:31.819556 7478 raft_consensus.cc:388] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:31.819587 7478 raft_consensus.cc:482] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:31.819640 7478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:31.820644 7478 raft_consensus.cc:504] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde746cc9de43b0b955f0413ed810b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 19:30:31.820729 7478 leader_election.cc:302] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edde746cc9de43b0b955f0413ed810b4; no voters:
I20260924 19:30:31.820901 7478 leader_election.cc:288] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:31.820921 7483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:31.821022 7483 raft_consensus.cc:686] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [term 1 LEADER]: Becoming Leader. State: Replica: edde746cc9de43b0b955f0413ed810b4, State: Running, Role: LEADER
I20260924 19:30:31.821102 7483 consensus_queue.cc:237] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde746cc9de43b0b955f0413ed810b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } }
I20260924 19:30:31.821146 7478 sys_catalog.cc:564] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:31.821413 7484 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edde746cc9de43b0b955f0413ed810b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde746cc9de43b0b955f0413ed810b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } } }
I20260924 19:30:31.821499 7484 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:31.821638 7485 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edde746cc9de43b0b955f0413ed810b4. Latest consensus state: current_term: 1 leader_uuid: "edde746cc9de43b0b955f0413ed810b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde746cc9de43b0b955f0413ed810b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34407 } } }
I20260924 19:30:31.821708 7485 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:31.821852 7488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:31.822306 7488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:31.824800 7488 catalog_manager.cc:1347] Generated new cluster ID: 73ef3738eda143cf84e897a1dab7a40c
I20260924 19:30:31.824849 7488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:30:31 dist-test-slave-r8cc krb5kdc[7397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:30:31.829562 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:31.818393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56569 (local address 127.0.0.1:34407)
0924 19:30:31.818553 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:31.818560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:31.818583 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:30:31.819089 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:31.819094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:31.819137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:31.819245 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:31.820058 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:31.821141 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:31.822311 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:31.822949 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:31.823058 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:31.825762 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:31.825774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:31.825774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:31.825786 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:31.827754 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:31.828180 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:31.828183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:31.828184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:31.828332 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:31.828521 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:31.828522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:31.828523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:31.828851 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:31.828998 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:31.829074 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:30:31.829206 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260924 19:30:31.830760 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:31.839568 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:31.832132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45409 (local address 127.0.0.1:34407)
0924 19:30:31.832168 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:31.832170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:31.832299 (+ 129us) server_negotiation.cc:407] Connection header received
0924 19:30:31.832396 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:31.832396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:31.832406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:31.832435 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:31.833149 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:31.833857 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:31.834937 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:31.835262 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:31.835304 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:31.836813 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:31.836814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:31.836814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:31.836822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:31.838336 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:31.838719 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:31.838720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:31.838720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:31.838794 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:31.839005 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:31.839005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:31.839006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:31.839122 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:31.839199 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:31.839397 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 19:30:31.839473 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:30:31.851444 7488 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:31.852151 7488 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:31.856637 7488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edde746cc9de43b0b955f0413ed810b4: Generated new TSK 0
I20260924 19:30:31.856833 7488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:31.861830 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7410
2026-09-24T19:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:31 dist-test-slave-r8cc krb5kdc[7509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:31 dist-test-slave-r8cc krb5kdc[7509](info): set up 2 sockets
Sep 24 19:30:31 dist-test-slave-r8cc krb5kdc[7509](info): commencing operation
krb5kdc: starting...
W20260924 19:30:32.876189 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:32 dist-test-slave-r8cc krb5kdc[7509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:32.988896 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:33.068540 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:33.068728 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:33.068750 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:33.069772 7522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:33.069802 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:33.069815 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:33.069828 7522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:33.069839 7522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:33.071185 7522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:33.071369 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:33.071521 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:33.076290 7527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:33.076339 7528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:33.076393 7522 server_base.cc:1034] running on GCE node
W20260924 19:30:33.076356 7530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:33.076721 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:33.076973 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:33.078106 7522 hybrid_clock.cc:648] HybridClock initialized: now 1790278233078088 us; error 34 us; skew 500 ppm
Sep 24 19:30:33 dist-test-slave-r8cc krb5kdc[7509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:33.080530 7522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:33.080968 7522 webserver.cc:460] Webserver started at http://127.0.0.1:41721/ using document root <none> and password file <none>
I20260924 19:30:33.081188 7522 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:33.081233 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:33.081393 7522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:33.082432 7522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "d01dc33f817143608ec39d4065879584"
format_stamp: "Formatted at 2026-09-24 19:30:33 on dist-test-slave-r8cc"
server_key: "eeba2a303579671b9d71deed79da7887"
server_key_iv: "72a83b2ee6fe972ac996231e750a979e"
server_key_version: "encryptionkey@0"
I20260924 19:30:33.082808 7522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "d01dc33f817143608ec39d4065879584"
format_stamp: "Formatted at 2026-09-24 19:30:33 on dist-test-slave-r8cc"
server_key: "eeba2a303579671b9d71deed79da7887"
server_key_iv: "72a83b2ee6fe972ac996231e750a979e"
server_key_version: "encryptionkey@0"
I20260924 19:30:33.084100 7522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:30:33.084872 7537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:33.085052 7522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:33.085140 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "d01dc33f817143608ec39d4065879584"
format_stamp: "Formatted at 2026-09-24 19:30:33 on dist-test-slave-r8cc"
server_key: "eeba2a303579671b9d71deed79da7887"
server_key_iv: "72a83b2ee6fe972ac996231e750a979e"
server_key_version: "encryptionkey@0"
I20260924 19:30:33.085214 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:33.110720 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:33.112709 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:33.112870 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:33.116742 7522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39453
I20260924 19:30:33.116830 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39453 every 8 connection(s)
I20260924 19:30:33.117178 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:33.118072 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:33.120352 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584: Bootstrap starting.
I20260924 19:30:33.120877 7590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:33.121096 7590 log.cc:824] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:33.121717 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584: No bootstrap required, opened a new log
I20260924 19:30:33.123275 7590 raft_consensus.cc:348] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01dc33f817143608ec39d4065879584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260924 19:30:33.123405 7590 raft_consensus.cc:374] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:33.123425 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7522
I20260924 19:30:33.123427 7590 raft_consensus.cc:729] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01dc33f817143608ec39d4065879584, State: Initialized, Role: FOLLOWER
I20260924 19:30:33.123559 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:33.123594 7590 consensus_queue.cc:260] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01dc33f817143608ec39d4065879584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260924 19:30:33.123675 7590 raft_consensus.cc:388] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:33.123735 7590 raft_consensus.cc:482] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:33.123780 7590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:33.123786 19350 external_mini_cluster.cc:1428] Setting key c490001a1f534d31b75bf4c753f052ad
I20260924 19:30:33.124456 7590 raft_consensus.cc:504] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01dc33f817143608ec39d4065879584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260924 19:30:33.124580 7590 leader_election.cc:302] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d01dc33f817143608ec39d4065879584; no voters:
I20260924 19:30:33.124727 7590 leader_election.cc:288] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:33.124879 7596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:33.124974 7590 sys_catalog.cc:564] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:33.125015 7596 raft_consensus.cc:686] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [term 1 LEADER]: Becoming Leader. State: Replica: d01dc33f817143608ec39d4065879584, State: Running, Role: LEADER
I20260924 19:30:33.125375 7596 consensus_queue.cc:237] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01dc33f817143608ec39d4065879584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260924 19:30:33.125830 7597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d01dc33f817143608ec39d4065879584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01dc33f817143608ec39d4065879584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } } }
I20260924 19:30:33.125919 7597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:33.126111 7595 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d01dc33f817143608ec39d4065879584. Latest consensus state: current_term: 1 leader_uuid: "d01dc33f817143608ec39d4065879584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d01dc33f817143608ec39d4065879584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } } }
I20260924 19:30:33.126420 7595 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584 [sys.catalog]: This master's current role is: LEADER
W20260924 19:30:33.126857 7611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:33.126919 7611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:33.126973 7605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:33.127164 7605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:33.128595 7605 catalog_manager.cc:1347] Generated new cluster ID: 8cc11e152c014f9aaa68a963ecf9076d
I20260924 19:30:33.128649 7605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:33.129931 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:33.124582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50809 (local address 127.0.0.1:39453)
0924 19:30:33.124787 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:33.124808 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:33.124844 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:30:33.124965 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:33.124974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:33.125038 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:33.125206 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:33.125466 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:33.126898 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:33.127874 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:33.128535 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:33.128647 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:33.129319 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:33.129322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:33.129323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:33.129516 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:33.129551 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:33.129565 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:33.129628 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:33.129682 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:30:33.129742 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260924 19:30:33.130650 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:33.135309 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:33.131909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57491 (local address 127.0.0.1:39453)
0924 19:30:33.131944 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:33.131946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:33.132060 (+ 114us) server_negotiation.cc:407] Connection header received
0924 19:30:33.132088 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:33.132089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:33.132100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:33.132169 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:33.132310 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:33.133051 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:33.134032 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:33.134324 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:33.134380 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:33.135091 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:33.135092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:33.135092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:33.135096 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:33.135106 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:33.135110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:33.135149 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:33.135213 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:30:33.135234 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:33.152691 7605 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:33.153268 7605 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:33.161481 7605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d01dc33f817143608ec39d4065879584: Generated new TSK 0
I20260924 19:30:33.161679 7605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:33.171978 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7522
2026-09-24T19:30:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:33 dist-test-slave-r8cc krb5kdc[7621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:33 dist-test-slave-r8cc krb5kdc[7621](info): set up 2 sockets
Sep 24 19:30:33 dist-test-slave-r8cc krb5kdc[7621](info): commencing operation
krb5kdc: starting...
W20260924 19:30:34.196070 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:34 dist-test-slave-r8cc krb5kdc[7621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:34.309082 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:34.389307 7634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:34.389515 7634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:34.389539 7634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:34.390617 7634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:34.390653 7634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:34.390668 7634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:34.390681 7634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:34.390693 7634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:34.392102 7634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:34.392316 7634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:34.392477 7634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:34.397079 7640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:34.397152 7634 server_base.cc:1034] running on GCE node
W20260924 19:30:34.397068 7639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:34.397066 7642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:34.397567 7634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:34.397853 7634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:34.399013 7634 hybrid_clock.cc:648] HybridClock initialized: now 1790278234399007 us; error 32 us; skew 500 ppm
Sep 24 19:30:34 dist-test-slave-r8cc krb5kdc[7621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:34.405131 7634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:34.405602 7634 webserver.cc:460] Webserver started at http://127.0.0.1:41725/ using document root <none> and password file <none>
I20260924 19:30:34.405879 7634 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:34.405951 7634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:34.406098 7634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:34.407263 7634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "56f5a6a9fe2748ce94fc75df14099937"
format_stamp: "Formatted at 2026-09-24 19:30:34 on dist-test-slave-r8cc"
server_key: "f7c71e4f0bac5680ca2b4026877142f9"
server_key_iv: "e716b7134a271fbda3a5b882aaf096f0"
server_key_version: "encryptionkey@0"
I20260924 19:30:34.407652 7634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "56f5a6a9fe2748ce94fc75df14099937"
format_stamp: "Formatted at 2026-09-24 19:30:34 on dist-test-slave-r8cc"
server_key: "f7c71e4f0bac5680ca2b4026877142f9"
server_key_iv: "e716b7134a271fbda3a5b882aaf096f0"
server_key_version: "encryptionkey@0"
I20260924 19:30:34.409168 7634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:34.409888 7649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:34.410066 7634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:34.410177 7634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "56f5a6a9fe2748ce94fc75df14099937"
format_stamp: "Formatted at 2026-09-24 19:30:34 on dist-test-slave-r8cc"
server_key: "f7c71e4f0bac5680ca2b4026877142f9"
server_key_iv: "e716b7134a271fbda3a5b882aaf096f0"
server_key_version: "encryptionkey@0"
I20260924 19:30:34.410269 7634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:34.413913 7634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:34.415805 7634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:34.415982 7634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:34.419922 7634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33855
I20260924 19:30:34.419950 7701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33855 every 8 connection(s)
I20260924 19:30:34.420336 7634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:34.421168 7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:34.423246 7702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937: Bootstrap starting.
I20260924 19:30:34.423414 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7634
I20260924 19:30:34.423547 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:34.423758 19350 external_mini_cluster.cc:1428] Setting key dded346521867caae0016a0cad5b68d3
I20260924 19:30:34.423929 7702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:34.424306 7702 log.cc:824] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:34.425072 7702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937: No bootstrap required, opened a new log
I20260924 19:30:34.426399 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:34.424275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59813 (local address 127.0.0.1:33855)
0924 19:30:34.424547 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:34.424556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:34.424587 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:30:34.425139 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:34.425147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:34.425186 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:34.425329 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:34.426005 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:34.426009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:34.426009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:34.426013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:34.426052 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:34.426065 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:34.426148 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:34.426179 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 19:30:34.426215 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260924 19:30:34.427295 7702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5a6a9fe2748ce94fc75df14099937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260924 19:30:34.427412 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:34.427448 7702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:34.427480 7702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f5a6a9fe2748ce94fc75df14099937, State: Initialized, Role: FOLLOWER
I20260924 19:30:34.427588 7702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5a6a9fe2748ce94fc75df14099937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260924 19:30:34.427676 7702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:34.427702 7702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:34.427762 7702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:34.428463 7702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5a6a9fe2748ce94fc75df14099937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260924 19:30:34.428582 7702 leader_election.cc:302] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f5a6a9fe2748ce94fc75df14099937; no voters:
I20260924 19:30:34.428778 7702 leader_election.cc:288] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:34.428799 7711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:34.429054 7702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:34.429352 7711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [term 1 LEADER]: Becoming Leader. State: Replica: 56f5a6a9fe2748ce94fc75df14099937, State: Running, Role: LEADER
I20260924 19:30:34.429481 7711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5a6a9fe2748ce94fc75df14099937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260924 19:30:34.429855 7711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f5a6a9fe2748ce94fc75df14099937. Latest consensus state: current_term: 1 leader_uuid: "56f5a6a9fe2748ce94fc75df14099937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5a6a9fe2748ce94fc75df14099937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } } }
I20260924 19:30:34.430312 7711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:34.430401 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:34.429045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53085 (local address 127.0.0.1:33855)
0924 19:30:34.429077 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:34.429079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:34.429288 (+ 209us) server_negotiation.cc:407] Connection header received
0924 19:30:34.429308 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:34.429309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:34.429314 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:34.429355 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:34.430050 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:34.430052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:34.430052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:34.430057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:34.430072 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:34.430077 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:34.430122 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:34.430244 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 19:30:34.430256 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:30:34.430217 7712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56f5a6a9fe2748ce94fc75df14099937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5a6a9fe2748ce94fc75df14099937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } } }
I20260924 19:30:34.430578 7712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:34.430686 7725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:34.431337 7725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:34.432813 7725 catalog_manager.cc:1347] Generated new cluster ID: 825cfca6527d4de9bc976ba9dc4c5c40
I20260924 19:30:34.432864 7725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:34.443816 7725 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:34.443964 7725 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:34.448580 7725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56f5a6a9fe2748ce94fc75df14099937: Generated new TSK 0
I20260924 19:30:34.448792 7725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:34.455777 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7634
2026-09-24T19:30:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:34 dist-test-slave-r8cc krb5kdc[7733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:34 dist-test-slave-r8cc krb5kdc[7733](info): set up 2 sockets
Sep 24 19:30:34 dist-test-slave-r8cc krb5kdc[7733](info): commencing operation
krb5kdc: starting...
W20260924 19:30:35.492272 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:35 dist-test-slave-r8cc krb5kdc[7733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:35.605202 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:35.684314 7746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:35.684508 7746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:35.684533 7746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:35.685596 7746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:35.685631 7746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:35.685645 7746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:35.685671 7746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:35.685683 7746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:35.687108 7746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:35.687294 7746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:35.687461 7746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:35.692144 7754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:35.692140 7752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:35.692178 7751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:35.692451 7746 server_base.cc:1034] running on GCE node
I20260924 19:30:35.692683 7746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:35.692966 7746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:35.694132 7746 hybrid_clock.cc:648] HybridClock initialized: now 1790278235694116 us; error 26 us; skew 500 ppm
Sep 24 19:30:35 dist-test-slave-r8cc krb5kdc[7733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:35.696659 7746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:35.697067 7746 webserver.cc:460] Webserver started at http://127.0.0.1:45699/ using document root <none> and password file <none>
I20260924 19:30:35.697307 7746 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:35.697371 7746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:35.697505 7746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:35.698529 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "34fa5137e2c844139d0cbbbdd3ff67f7"
format_stamp: "Formatted at 2026-09-24 19:30:35 on dist-test-slave-r8cc"
server_key: "bf9cb460805383803d7a304bfec91a73"
server_key_iv: "f3ec6a38a64e32d65fc6e4325a79bd58"
server_key_version: "encryptionkey@0"
I20260924 19:30:35.698871 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "34fa5137e2c844139d0cbbbdd3ff67f7"
format_stamp: "Formatted at 2026-09-24 19:30:35 on dist-test-slave-r8cc"
server_key: "bf9cb460805383803d7a304bfec91a73"
server_key_iv: "f3ec6a38a64e32d65fc6e4325a79bd58"
server_key_version: "encryptionkey@0"
I20260924 19:30:35.700176 7746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:30:35.700822 7761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:35.700981 7746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:35.701076 7746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "34fa5137e2c844139d0cbbbdd3ff67f7"
format_stamp: "Formatted at 2026-09-24 19:30:35 on dist-test-slave-r8cc"
server_key: "bf9cb460805383803d7a304bfec91a73"
server_key_iv: "f3ec6a38a64e32d65fc6e4325a79bd58"
server_key_version: "encryptionkey@0"
I20260924 19:30:35.701154 7746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:35.725005 7746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:35.727341 7746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:35.727514 7746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:35.731406 7746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43005
I20260924 19:30:35.731451 7813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43005 every 8 connection(s)
I20260924 19:30:35.731817 7746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:35.732679 7814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:35.734846 7814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7: Bootstrap starting.
I20260924 19:30:35.735356 7814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:35.735587 7814 log.cc:824] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:35.736258 7814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7: No bootstrap required, opened a new log
I20260924 19:30:35.737749 7814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260924 19:30:35.737890 7814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:35.737911 7814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34fa5137e2c844139d0cbbbdd3ff67f7, State: Initialized, Role: FOLLOWER
I20260924 19:30:35.738044 7814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260924 19:30:35.738118 7814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:35.738139 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:35.738175 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:35.738729 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260924 19:30:35.738821 7814 leader_election.cc:302] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34fa5137e2c844139d0cbbbdd3ff67f7; no voters:
I20260924 19:30:35.738947 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:35.738992 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:35.739153 7817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [term 1 LEADER]: Becoming Leader. State: Replica: 34fa5137e2c844139d0cbbbdd3ff67f7, State: Running, Role: LEADER
I20260924 19:30:35.739183 7814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:35.739291 7817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260924 19:30:35.739575 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7746
I20260924 19:30:35.739693 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:35.739796 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } } }
I20260924 19:30:35.739884 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:35.739893 19350 external_mini_cluster.cc:1428] Setting key 95b69e4aaa79a9aa17501a61d4e33059
I20260924 19:30:35.739928 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34fa5137e2c844139d0cbbbdd3ff67f7. Latest consensus state: current_term: 1 leader_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34fa5137e2c844139d0cbbbdd3ff67f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } } }
I20260924 19:30:35.739996 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:35.740716 7826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:35.741248 7826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:35.742738 7826 catalog_manager.cc:1347] Generated new cluster ID: 4781dec5fd724cb8af18ec3b7b9073a7
I20260924 19:30:35.742800 7826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:30:35 dist-test-slave-r8cc krb5kdc[7733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:30:35.751720 7826 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:35.751752 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:35.740663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:35829 (local address 127.0.0.1:43005)
0924 19:30:35.740853 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:35.740862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:35.740897 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:30:35.740988 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:35.740994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:35.741072 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:35.741795 (+ 723us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:35.741972 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:35.743276 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:35.744371 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:35.744957 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:35.745050 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:35.747808 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:35.747813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:35.747813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:35.747825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:35.749860 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:35.750293 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:35.750296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:35.750297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:35.750407 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:35.750649 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:35.750650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:35.750650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:35.750958 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:35.751129 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:35.751260 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:30:35.751368 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260924 19:30:35.752404 7826 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:35.752676 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:35.755342 7826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34fa5137e2c844139d0cbbbdd3ff67f7: Generated new TSK 0
I20260924 19:30:35.755486 7826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:35.760627 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:35.754121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:33491 (local address 127.0.0.1:43005)
0924 19:30:35.754158 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:35.754160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:35.754281 (+ 121us) server_negotiation.cc:407] Connection header received
0924 19:30:35.754337 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:35.754338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:35.754347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:35.754399 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:35.755012 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:35.755713 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:35.756740 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:35.756934 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:35.756983 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:35.758230 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:35.758231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:35.758231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:35.758237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:35.759489 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:35.759875 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:35.759876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:35.759876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:35.759933 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:35.760207 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:35.760208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:35.760208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:35.760304 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:35.760381 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:35.760529 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 19:30:35.760574 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:35.762349 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7746
W20260924 19:30:35.766523 7731 connection.cc:535] client connection to 127.0.0.1:43005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43005): Connection reset by peer (error 104)
2026-09-24T19:30:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:35 dist-test-slave-r8cc krb5kdc[7845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:35 dist-test-slave-r8cc krb5kdc[7845](info): set up 2 sockets
Sep 24 19:30:35 dist-test-slave-r8cc krb5kdc[7845](info): commencing operation
krb5kdc: starting...
W20260924 19:30:36.792367 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:36 dist-test-slave-r8cc krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:36.904836 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:36.984376 7858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:36.984573 7858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:36.984596 7858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:36.985669 7858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:36.985705 7858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:36.985719 7858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:36.985733 7858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:36.985750 7858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:36.987125 7858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:36.987317 7858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:36.987480 7858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:36.992149 7864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:36.992159 7866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:36.992163 7863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:36.992259 7858 server_base.cc:1034] running on GCE node
I20260924 19:30:36.992614 7858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:36.992900 7858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:36.994064 7858 hybrid_clock.cc:648] HybridClock initialized: now 1790278236994052 us; error 44 us; skew 500 ppm
Sep 24 19:30:36 dist-test-slave-r8cc krb5kdc[7845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:36.996439 7858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:36.996841 7858 webserver.cc:460] Webserver started at http://127.0.0.1:39441/ using document root <none> and password file <none>
I20260924 19:30:36.997071 7858 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:36.997136 7858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:36.997296 7858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:36.998344 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "2da7b19334e24100b7104b9830530712"
format_stamp: "Formatted at 2026-09-24 19:30:36 on dist-test-slave-r8cc"
server_key: "239d60e4d98024cb9ea68fb3882d8d08"
server_key_iv: "d322ec039ff070e4400d473d37709f65"
server_key_version: "encryptionkey@0"
I20260924 19:30:36.998692 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "2da7b19334e24100b7104b9830530712"
format_stamp: "Formatted at 2026-09-24 19:30:36 on dist-test-slave-r8cc"
server_key: "239d60e4d98024cb9ea68fb3882d8d08"
server_key_iv: "d322ec039ff070e4400d473d37709f65"
server_key_version: "encryptionkey@0"
I20260924 19:30:36.999970 7858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:37.000730 7873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:37.000898 7858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:37.000993 7858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "2da7b19334e24100b7104b9830530712"
format_stamp: "Formatted at 2026-09-24 19:30:36 on dist-test-slave-r8cc"
server_key: "239d60e4d98024cb9ea68fb3882d8d08"
server_key_iv: "d322ec039ff070e4400d473d37709f65"
server_key_version: "encryptionkey@0"
I20260924 19:30:37.001075 7858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:37.009368 7858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:37.011029 7858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:37.011180 7858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:37.015094 7858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41819
I20260924 19:30:37.015122 7925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41819 every 8 connection(s)
I20260924 19:30:37.015460 7858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:37.016366 7926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:37.018482 7926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712: Bootstrap starting.
I20260924 19:30:37.018983 7926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:37.019248 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7858
I20260924 19:30:37.019323 7926 log.cc:824] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:37.019374 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:37.019593 19350 external_mini_cluster.cc:1428] Setting key 09b74acef3aa0ee1b48ca599a207a722
I20260924 19:30:37.020419 7926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712: No bootstrap required, opened a new log
I20260924 19:30:37.023090 7926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da7b19334e24100b7104b9830530712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260924 19:30:37.023245 7926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:37.023294 7926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da7b19334e24100b7104b9830530712, State: Initialized, Role: FOLLOWER
I20260924 19:30:37.023416 7926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da7b19334e24100b7104b9830530712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260924 19:30:37.023511 7926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:37.023569 7926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:37.023622 7926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:37.024223 7926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da7b19334e24100b7104b9830530712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260924 19:30:37.024336 7926 leader_election.cc:302] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da7b19334e24100b7104b9830530712; no voters:
I20260924 19:30:37.024494 7926 leader_election.cc:288] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:37.024538 7931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:37.024713 7931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [term 1 LEADER]: Becoming Leader. State: Replica: 2da7b19334e24100b7104b9830530712, State: Running, Role: LEADER
I20260924 19:30:37.025025 7931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da7b19334e24100b7104b9830530712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260924 19:30:37.024776 7926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:37.025769 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:37.020162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:57447 (local address 127.0.0.1:41819)
0924 19:30:37.020363 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:37.020371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:37.020402 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:30:37.020909 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:37.020918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:37.020980 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:37.021131 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:37.021302 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:37.022704 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:37.023569 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:37.024022 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:37.024208 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:37.024949 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:37.024953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:37.024954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:37.025144 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:37.025187 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:37.025200 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:37.025301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:37.025532 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:30:37.025560 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260924 19:30:37.026297 7933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da7b19334e24100b7104b9830530712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da7b19334e24100b7104b9830530712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260924 19:30:37.026345 7932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da7b19334e24100b7104b9830530712. Latest consensus state: current_term: 1 leader_uuid: "2da7b19334e24100b7104b9830530712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da7b19334e24100b7104b9830530712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260924 19:30:37.026412 7933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:37.026432 7932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:37.026682 7944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:37.026913 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:37.027206 7944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:37.028888 7944 catalog_manager.cc:1347] Generated new cluster ID: 394c6e1c2e434fb3aaf446fe8808c880
I20260924 19:30:37.028936 7944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:37.031911 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:37.028492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:32779 (local address 127.0.0.1:41819)
0924 19:30:37.028529 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:37.028531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:37.028681 (+ 150us) server_negotiation.cc:407] Connection header received
0924 19:30:37.028737 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:37.028738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:37.028755 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:37.028801 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:37.028995 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:37.029679 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:37.030615 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:37.030921 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:37.030995 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:37.031670 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:37.031671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:37.031672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:37.031676 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:37.031690 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:37.031694 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:37.031767 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:37.031829 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 19:30:37.031839 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:30:37.048210 7944 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:37.048723 7944 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:37.051862 7944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da7b19334e24100b7104b9830530712: Generated new TSK 0
I20260924 19:30:37.052054 7944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:37.056349 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7858
W20260924 19:30:37.060544 7843 connection.cc:535] client connection to 127.0.0.1:41819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41819): Connection reset by peer (error 104)
2026-09-24T19:30:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:37 dist-test-slave-r8cc krb5kdc[7957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:37 dist-test-slave-r8cc krb5kdc[7957](info): set up 2 sockets
Sep 24 19:30:37 dist-test-slave-r8cc krb5kdc[7957](info): commencing operation
krb5kdc: starting...
W20260924 19:30:38.076040 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:38 dist-test-slave-r8cc krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:38.188851 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:38.268051 7970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:38.268273 7970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:38.268296 7970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:38.269536 7970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:38.269588 7970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:38.269603 7970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:38.269654 7970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:38.269666 7970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:38.271185 7970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:38.271423 7970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:38.271677 7970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:30:38.276533 7970 server_base.cc:1034] running on GCE node
W20260924 19:30:38.276540 7978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:38.276554 7975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:38.276648 7976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:38.276939 7970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:38.277230 7970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:38.278399 7970 hybrid_clock.cc:648] HybridClock initialized: now 1790278238278391 us; error 36 us; skew 500 ppm
Sep 24 19:30:38 dist-test-slave-r8cc krb5kdc[7957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:38.280817 7970 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:38.281226 7970 webserver.cc:460] Webserver started at http://127.0.0.1:33613/ using document root <none> and password file <none>
I20260924 19:30:38.281472 7970 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:38.281540 7970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:38.281688 7970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:38.282721 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c"
format_stamp: "Formatted at 2026-09-24 19:30:38 on dist-test-slave-r8cc"
server_key: "78f9a9329aba9303ab2af5c1675732d1"
server_key_iv: "d449fbf6b4a8ffb1d77799a755755f44"
server_key_version: "encryptionkey@0"
I20260924 19:30:38.283069 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c"
format_stamp: "Formatted at 2026-09-24 19:30:38 on dist-test-slave-r8cc"
server_key: "78f9a9329aba9303ab2af5c1675732d1"
server_key_iv: "d449fbf6b4a8ffb1d77799a755755f44"
server_key_version: "encryptionkey@0"
I20260924 19:30:38.284412 7970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:38.285123 7985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:38.285295 7970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:38.285399 7970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c"
format_stamp: "Formatted at 2026-09-24 19:30:38 on dist-test-slave-r8cc"
server_key: "78f9a9329aba9303ab2af5c1675732d1"
server_key_iv: "d449fbf6b4a8ffb1d77799a755755f44"
server_key_version: "encryptionkey@0"
I20260924 19:30:38.285472 7970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:38.288697 7970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:38.291097 7970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:38.291240 7970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:38.295060 7970 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40893
I20260924 19:30:38.295089 8037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40893 every 8 connection(s)
I20260924 19:30:38.295415 7970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:38.296284 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:38.298427 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c: Bootstrap starting.
I20260924 19:30:38.298981 8038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:38.299254 8038 log.cc:824] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:38.299959 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c: No bootstrap required, opened a new log
I20260924 19:30:38.301585 8038 raft_consensus.cc:348] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260924 19:30:38.301759 8038 raft_consensus.cc:374] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:38.301781 8038 raft_consensus.cc:729] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16e4b8c302444d1ac71f6a2d3d06f4c, State: Initialized, Role: FOLLOWER
I20260924 19:30:38.301961 8038 consensus_queue.cc:260] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260924 19:30:38.302048 8038 raft_consensus.cc:388] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:38.302098 8038 raft_consensus.cc:482] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:38.302151 8038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:38.302690 8038 raft_consensus.cc:504] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260924 19:30:38.302802 8038 leader_election.cc:302] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a16e4b8c302444d1ac71f6a2d3d06f4c; no voters:
I20260924 19:30:38.302991 8038 leader_election.cc:288] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:38.303031 8041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:38.303194 8041 raft_consensus.cc:686] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [term 1 LEADER]: Becoming Leader. State: Replica: a16e4b8c302444d1ac71f6a2d3d06f4c, State: Running, Role: LEADER
I20260924 19:30:38.303260 8038 sys_catalog.cc:564] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:38.303326 8041 consensus_queue.cc:237] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260924 19:30:38.303377 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 7970
I20260924 19:30:38.303490 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:38.303694 19350 external_mini_cluster.cc:1428] Setting key 52d38318b090b9298100dfeb4d7d18fb
I20260924 19:30:38.303907 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } } }
I20260924 19:30:38.303993 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:38.304483 8043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a16e4b8c302444d1ac71f6a2d3d06f4c. Latest consensus state: current_term: 1 leader_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16e4b8c302444d1ac71f6a2d3d06f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } } }
I20260924 19:30:38.304579 8043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c [sys.catalog]: This master's current role is: LEADER
W20260924 19:30:38.305233 8060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:38.305330 8060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:38.305845 8050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:38.306047 8050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:38.306341 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:38.304451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:33889 (local address 127.0.0.1:40893)
0924 19:30:38.304668 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:38.304677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:38.304712 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:30:38.304854 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:38.304861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:38.304993 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:38.305148 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:38.305860 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:38.305864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:38.305864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:38.305868 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:38.305913 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:38.305927 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:38.306027 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:38.306122 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:30:38.306166 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260924 19:30:38.307224 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:38.307530 8050 catalog_manager.cc:1347] Generated new cluster ID: c60486bb5e9340ae9f4e277f8a8a0a0b
I20260924 19:30:38.307569 8050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:38.309810 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:38.308794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:44145 (local address 127.0.0.1:40893)
0924 19:30:38.308832 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:38.308833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:38.308959 (+ 126us) server_negotiation.cc:407] Connection header received
0924 19:30:38.308990 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:38.308991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:38.308996 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:38.309065 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:38.309596 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:38.309597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:38.309598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:38.309609 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:38.309624 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:38.309628 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:38.309690 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:38.309740 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:30:38.309751 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:30:38.329788 8050 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:38.330008 8050 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:38.335513 8050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a16e4b8c302444d1ac71f6a2d3d06f4c: Generated new TSK 0
I20260924 19:30:38.335700 8050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:38.343968 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 7970
2026-09-24T19:30:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:38 dist-test-slave-r8cc krb5kdc[8070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:38 dist-test-slave-r8cc krb5kdc[8070](info): set up 2 sockets
Sep 24 19:30:38 dist-test-slave-r8cc krb5kdc[8070](info): commencing operation
krb5kdc: starting...
W20260924 19:30:39.368216 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:39 dist-test-slave-r8cc krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:39.478818 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:39.557415 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:39.557605 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:39.557627 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:39.558760 8083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:39.558800 8083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:39.558815 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:39.558827 8083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:39.558840 8083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:39.560789 8083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:39.561036 8083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:39.561266 8083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:39.566061 8091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:39.566092 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:39.566092 8088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:39.566145 8083 server_base.cc:1034] running on GCE node
I20260924 19:30:39.566490 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:39.566815 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:39.567973 8083 hybrid_clock.cc:648] HybridClock initialized: now 1790278239567963 us; error 40 us; skew 500 ppm
Sep 24 19:30:39 dist-test-slave-r8cc krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:39.573890 8083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:39.574294 8083 webserver.cc:460] Webserver started at http://127.0.0.1:41427/ using document root <none> and password file <none>
I20260924 19:30:39.574517 8083 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:39.574581 8083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:39.574726 8083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:39.575774 8083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "d2e452e092814d26aefaafe506f07efe"
format_stamp: "Formatted at 2026-09-24 19:30:39 on dist-test-slave-r8cc"
server_key: "1aa0aa4e5feca0844f4dfa20afe28c3f"
server_key_iv: "12b537fcc5470fe24278b3804d901428"
server_key_version: "encryptionkey@0"
I20260924 19:30:39.576185 8083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "d2e452e092814d26aefaafe506f07efe"
format_stamp: "Formatted at 2026-09-24 19:30:39 on dist-test-slave-r8cc"
server_key: "1aa0aa4e5feca0844f4dfa20afe28c3f"
server_key_iv: "12b537fcc5470fe24278b3804d901428"
server_key_version: "encryptionkey@0"
I20260924 19:30:39.577507 8083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:39.578238 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:39.578404 8083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:39.578498 8083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "d2e452e092814d26aefaafe506f07efe"
format_stamp: "Formatted at 2026-09-24 19:30:39 on dist-test-slave-r8cc"
server_key: "1aa0aa4e5feca0844f4dfa20afe28c3f"
server_key_iv: "12b537fcc5470fe24278b3804d901428"
server_key_version: "encryptionkey@0"
I20260924 19:30:39.578580 8083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:39.612501 8083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:39.614315 8083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:39.614421 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:39.618167 8083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44199
I20260924 19:30:39.618242 8150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44199 every 8 connection(s)
I20260924 19:30:39.618562 8083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:39.619467 8151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:39.621762 8151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe: Bootstrap starting.
I20260924 19:30:39.622323 8151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:39.622602 8151 log.cc:824] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:39.623258 8151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe: No bootstrap required, opened a new log
I20260924 19:30:39.623454 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8083
I20260924 19:30:39.623569 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:39.623785 19350 external_mini_cluster.cc:1428] Setting key 308a806475c68aae6567d00a85c8a615
I20260924 19:30:39.625568 8151 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e452e092814d26aefaafe506f07efe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260924 19:30:39.625768 8151 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:39.625803 8151 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e452e092814d26aefaafe506f07efe, State: Initialized, Role: FOLLOWER
I20260924 19:30:39.625926 8151 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e452e092814d26aefaafe506f07efe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260924 19:30:39.626013 8151 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:39.626047 8151 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:39.626137 8151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:39.626730 8151 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e452e092814d26aefaafe506f07efe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260924 19:30:39.626837 8151 leader_election.cc:302] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e452e092814d26aefaafe506f07efe; no voters:
I20260924 19:30:39.626981 8151 leader_election.cc:288] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:39.627048 8156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:39.627206 8151 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:39.627233 8156 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [term 1 LEADER]: Becoming Leader. State: Replica: d2e452e092814d26aefaafe506f07efe, State: Running, Role: LEADER
I20260924 19:30:39.627415 8156 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e452e092814d26aefaafe506f07efe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260924 19:30:39.627774 8157 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e452e092814d26aefaafe506f07efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e452e092814d26aefaafe506f07efe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } } }
I20260924 19:30:39.627848 8157 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:39.628063 8158 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e452e092814d26aefaafe506f07efe. Latest consensus state: current_term: 1 leader_uuid: "d2e452e092814d26aefaafe506f07efe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e452e092814d26aefaafe506f07efe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } } }
I20260924 19:30:39.628439 8158 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe [sys.catalog]: This master's current role is: LEADER
W20260924 19:30:39.628925 8172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:39.630132 8172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:39.630200 8165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:39.630730 8165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:30:39 dist-test-slave-r8cc krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:30:39.632450 8165 catalog_manager.cc:1347] Generated new cluster ID: 2ac7015ef50f4572b287e056148a7334
I20260924 19:30:39.632510 8165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:39.637419 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:39.624301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:59811 (local address 127.0.0.1:44199)
0924 19:30:39.624489 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:39.624499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:39.624532 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:30:39.625783 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:39.625791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:39.625855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:39.626009 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:39.626700 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:39.628136 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:39.629197 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:39.629795 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:39.629901 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:39.632863 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:39.632867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:39.632867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:39.632879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:39.635329 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:39.635805 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:39.635808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:39.635809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:39.635908 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:39.636184 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:39.636185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:39.636186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:39.636486 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:39.636641 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:39.637123 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 19:30:39.637221 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260924 19:30:39.638351 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:39.647100 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:39.639800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:34529 (local address 127.0.0.1:44199)
0924 19:30:39.639838 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:39.639839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:39.639948 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:30:39.640037 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:39.640037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:39.640047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:39.640073 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:39.640792 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:39.641526 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:39.642547 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:39.642878 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:39.642918 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:39.644315 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:39.644316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:39.644317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:39.644323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:39.645853 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:39.646256 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:39.646257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:39.646257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:39.646304 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:39.646524 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:39.646525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:39.646525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:39.646641 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:39.646713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:39.646953 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:30:39.647013 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:30:39.651392 8165 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:39.651881 8165 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:39.657266 8165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e452e092814d26aefaafe506f07efe: Generated new TSK 0
I20260924 19:30:39.657428 8165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:39.659955 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8083
W20260924 19:30:39.664168 8068 connection.cc:535] client connection to 127.0.0.1:44199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44199): Connection reset by peer (error 104)
2026-09-24T19:30:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:39 dist-test-slave-r8cc krb5kdc[8182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:39 dist-test-slave-r8cc krb5kdc[8182](info): set up 2 sockets
Sep 24 19:30:39 dist-test-slave-r8cc krb5kdc[8182](info): commencing operation
krb5kdc: starting...
W20260924 19:30:40.692297 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:40 dist-test-slave-r8cc krb5kdc[8182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:40.804307 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:40.883545 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:40.883781 8195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:40.883821 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:40.885000 8195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:40.885046 8195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:40.885059 8195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:40.885072 8195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:40.885130 8195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:40.886600 8195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:40.886855 8195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:40.887100 8195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:40.891886 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:40.891955 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:40.891880 8200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:40.892042 8195 server_base.cc:1034] running on GCE node
I20260924 19:30:40.892316 8195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:40.892599 8195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:40.893741 8195 hybrid_clock.cc:648] HybridClock initialized: now 1790278240893716 us; error 42 us; skew 500 ppm
Sep 24 19:30:40 dist-test-slave-r8cc krb5kdc[8182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:40.896163 8195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:40.896584 8195 webserver.cc:460] Webserver started at http://127.0.0.1:33305/ using document root <none> and password file <none>
I20260924 19:30:40.896813 8195 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:40.896883 8195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:40.897032 8195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:40.898072 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502"
format_stamp: "Formatted at 2026-09-24 19:30:40 on dist-test-slave-r8cc"
server_key: "d2c18309aa25939b20e47bb96cc73987"
server_key_iv: "b91a45407125324d428d80dbc1f8d53d"
server_key_version: "encryptionkey@0"
I20260924 19:30:40.898429 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502"
format_stamp: "Formatted at 2026-09-24 19:30:40 on dist-test-slave-r8cc"
server_key: "d2c18309aa25939b20e47bb96cc73987"
server_key_iv: "b91a45407125324d428d80dbc1f8d53d"
server_key_version: "encryptionkey@0"
I20260924 19:30:40.899709 8195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:40.900449 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:40.900611 8195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:40.900717 8195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502"
format_stamp: "Formatted at 2026-09-24 19:30:40 on dist-test-slave-r8cc"
server_key: "d2c18309aa25939b20e47bb96cc73987"
server_key_iv: "b91a45407125324d428d80dbc1f8d53d"
server_key_version: "encryptionkey@0"
I20260924 19:30:40.900796 8195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:40.921106 8195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:40.922911 8195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:40.923074 8195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:40.926764 8195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45267
I20260924 19:30:40.926796 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45267 every 8 connection(s)
I20260924 19:30:40.927138 8195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:40.928023 8263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:40.928715 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8195
I20260924 19:30:40.928838 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:40.929051 19350 external_mini_cluster.cc:1428] Setting key f8eba923800fb9b10ace519346ed13ad
I20260924 19:30:40.930898 8263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502: Bootstrap starting.
I20260924 19:30:40.931473 8263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:40.931834 8263 log.cc:824] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:40.932636 8263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502: No bootstrap required, opened a new log
I20260924 19:30:40.934561 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:40.929560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:58857 (local address 127.0.0.1:45267)
0924 19:30:40.929722 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:40.929729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:40.929754 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:30:40.930139 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:40.930145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:40.930189 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:40.930310 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:40.930561 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:40.931465 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:40.932665 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:40.933108 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:40.933189 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:40.933920 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:40.933923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:40.933923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:40.934077 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:40.934106 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:40.934117 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:40.934179 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:40.934349 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 19:30:40.934382 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260924 19:30:40.934990 8263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260924 19:30:40.935166 8263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:40.935204 8263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc7cf9fdf1c4f91a55ceb56b491f502, State: Initialized, Role: FOLLOWER
I20260924 19:30:40.935345 8263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260924 19:30:40.935436 8263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:40.935479 8263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:40.935528 8263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:40.935576 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:40.936271 8263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260924 19:30:40.936388 8263 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc7cf9fdf1c4f91a55ceb56b491f502; no voters:
I20260924 19:30:40.936542 8263 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:40.936590 8268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:40.936781 8268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc7cf9fdf1c4f91a55ceb56b491f502, State: Running, Role: LEADER
I20260924 19:30:40.936800 8263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:40.936884 8268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260924 19:30:40.937229 8268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } } }
I20260924 19:30:40.937361 8268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:40.937523 8268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc7cf9fdf1c4f91a55ceb56b491f502. Latest consensus state: current_term: 1 leader_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7cf9fdf1c4f91a55ceb56b491f502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } } }
I20260924 19:30:40.937598 8268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:40.937743 8282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:40.938261 8282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:40.940685 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:40.937255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:48065 (local address 127.0.0.1:45267)
0924 19:30:40.937294 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:40.937296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:40.937378 (+ 82us) server_negotiation.cc:407] Connection header received
0924 19:30:40.937484 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:40.937485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:40.937497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:40.937528 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:40.937759 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:40.938505 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:40.939472 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:40.939660 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:40.939716 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:40.940432 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:40.940433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:40.940434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:40.940437 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:40.940449 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:40.940452 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:40.940498 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:40.940567 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:30:40.940575 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:30:40.944832 8282 catalog_manager.cc:1347] Generated new cluster ID: bb1d739ae81046aa96e78dcc7566cbed
I20260924 19:30:40.944895 8282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:40.962210 8282 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:40.962754 8282 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:40.968132 8282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc7cf9fdf1c4f91a55ceb56b491f502: Generated new TSK 0
I20260924 19:30:40.968328 8282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:40.977478 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8195
W20260924 19:30:40.981964 8180 connection.cc:535] client connection to 127.0.0.1:45267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45267): Connection reset by peer (error 104)
2026-09-24T19:30:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:41 dist-test-slave-r8cc krb5kdc[8294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:41 dist-test-slave-r8cc krb5kdc[8294](info): set up 2 sockets
Sep 24 19:30:41 dist-test-slave-r8cc krb5kdc[8294](info): commencing operation
krb5kdc: starting...
W20260924 19:30:41.996250 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:42 dist-test-slave-r8cc krb5kdc[8294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:30:42.107540 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:42.188297 8307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:42.188509 8307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:42.188544 8307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:42.189692 8307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:42.189761 8307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:42.189787 8307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:42.189819 8307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:42.189843 8307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:42.191354 8307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:42.191619 8307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:42.191836 8307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:42.196537 8313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:42.196568 8307 server_base.cc:1034] running on GCE node
W20260924 19:30:42.196552 8312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:42.196695 8315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:42.196983 8307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:42.197273 8307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:42.198407 8307 hybrid_clock.cc:648] HybridClock initialized: now 1790278242198400 us; error 29 us; skew 500 ppm
Sep 24 19:30:42 dist-test-slave-r8cc krb5kdc[8294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:42.200819 8307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:30:42.201227 8307 webserver.cc:460] Webserver started at http://127.0.0.1:41717/ using document root <none> and password file <none>
I20260924 19:30:42.201453 8307 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:42.201498 8307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:42.201586 8307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:42.202575 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "90212c7d28584411bb664bce80013b34"
format_stamp: "Formatted at 2026-09-24 19:30:42 on dist-test-slave-r8cc"
server_key: "e0b9795199b1b90741e0ecd01d1ad6d8"
server_key_iv: "40b991d45092bc6634e59e6d3baf8a35"
server_key_version: "encryptionkey@0"
I20260924 19:30:42.202865 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "90212c7d28584411bb664bce80013b34"
format_stamp: "Formatted at 2026-09-24 19:30:42 on dist-test-slave-r8cc"
server_key: "e0b9795199b1b90741e0ecd01d1ad6d8"
server_key_iv: "40b991d45092bc6634e59e6d3baf8a35"
server_key_version: "encryptionkey@0"
I20260924 19:30:42.204068 8307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:30:42.204784 8322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:42.204978 8307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:42.205049 8307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "90212c7d28584411bb664bce80013b34"
format_stamp: "Formatted at 2026-09-24 19:30:42 on dist-test-slave-r8cc"
server_key: "e0b9795199b1b90741e0ecd01d1ad6d8"
server_key_iv: "40b991d45092bc6634e59e6d3baf8a35"
server_key_version: "encryptionkey@0"
I20260924 19:30:42.205099 8307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:42.208168 8307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:42.210320 8307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:42.210417 8307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:42.214152 8307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36769
I20260924 19:30:42.214205 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36769 every 8 connection(s)
I20260924 19:30:42.214533 8307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:42.215379 8375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:42.217604 8375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34: Bootstrap starting.
I20260924 19:30:42.218168 8375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:42.218441 8375 log.cc:824] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:42.219097 8375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34: No bootstrap required, opened a new log
I20260924 19:30:42.220697 8375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90212c7d28584411bb664bce80013b34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36769 } }
I20260924 19:30:42.220835 8375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:42.220857 8375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90212c7d28584411bb664bce80013b34, State: Initialized, Role: FOLLOWER
I20260924 19:30:42.221024 8375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90212c7d28584411bb664bce80013b34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36769 } }
I20260924 19:30:42.221105 8375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:42.221155 8375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:42.221203 8375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:42.221721 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8307
I20260924 19:30:42.221743 8375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90212c7d28584411bb664bce80013b34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36769 } }
I20260924 19:30:42.221820 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:42.221860 8375 leader_election.cc:302] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90212c7d28584411bb664bce80013b34; no voters:
I20260924 19:30:42.222035 19350 external_mini_cluster.cc:1428] Setting key ca93537bb39b932d6bcac6fa3730fcf2
I20260924 19:30:42.222061 8375 leader_election.cc:288] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:42.222081 8378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:42.222244 8378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [term 1 LEADER]: Becoming Leader. State: Replica: 90212c7d28584411bb664bce80013b34, State: Running, Role: LEADER
I20260924 19:30:42.222309 8375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:42.222378 8378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90212c7d28584411bb664bce80013b34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36769 } }
I20260924 19:30:42.222828 8380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90212c7d28584411bb664bce80013b34. Latest consensus state: current_term: 1 leader_uuid: "90212c7d28584411bb664bce80013b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90212c7d28584411bb664bce80013b34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36769 } } }
I20260924 19:30:42.222965 8379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90212c7d28584411bb664bce80013b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90212c7d28584411bb664bce80013b34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36769 } } }
I20260924 19:30:42.222970 8380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:42.223070 8379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:42.223497 8389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:42.223997 8389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:42.225703 8389 catalog_manager.cc:1347] Generated new cluster ID: dd55dff9a6b1475286823a3662ae3946
I20260924 19:30:42.225802 8389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:42.226277 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:42.222634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:33135 (local address 127.0.0.1:36769)
0924 19:30:42.224764 (+ 2130us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:42.224772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:42.224798 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:30:42.224888 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:42.224894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:42.224919 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:42.225089 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:42.225817 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:42.225821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:42.225821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:42.225825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:42.225868 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:42.225882 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:42.225965 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:42.226048 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:30:42.226093 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":85,"threads_started":1}
I20260924 19:30:42.227167 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:42.233644 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:42.232509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.3:60677 (local address 127.0.0.1:36769)
0924 19:30:42.232580 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:42.232582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:42.232640 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:30:42.232725 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:42.232726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:42.232740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:42.232794 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:42.233422 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:42.233423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:42.233424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:42.233427 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:42.233442 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:42.233447 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:42.233497 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:42.233564 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:30:42.233578 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:30:42.239900 8389 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:42.240082 8389 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:42.248274 8389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90212c7d28584411bb664bce80013b34: Generated new TSK 0
I20260924 19:30:42.248497 8389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:42.250659 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8307
2026-09-24T19:30:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:42 dist-test-slave-r8cc krb5kdc[8406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:42 dist-test-slave-r8cc krb5kdc[8406](info): set up 2 sockets
Sep 24 19:30:42 dist-test-slave-r8cc krb5kdc[8406](info): commencing operation
krb5kdc: starting...
W20260924 19:30:43.264036 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:43 dist-test-slave-r8cc krb5kdc[8406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:43.374894 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:36539
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:43.455808 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:43.456058 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:43.456096 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:43.457939 8419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:43.458006 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:43.458031 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:43.458086 8419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:43.458120 8419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:43.460691 8419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:36539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:43.461107 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:43.461392 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:43.466238 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:43.466274 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:43.466333 8419 server_base.cc:1034] running on GCE node
W20260924 19:30:43.466427 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:43.466650 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:43.466950 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:43.468139 8419 hybrid_clock.cc:648] HybridClock initialized: now 1790278243468128 us; error 44 us; skew 500 ppm
Sep 24 19:30:43 dist-test-slave-r8cc krb5kdc[8406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278243, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:43.470458 8419 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:43.470852 8419 webserver.cc:460] Webserver started at http://127.18.229.190:44267/ using document root <none> and password file <none>
I20260924 19:30:43.471060 8419 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:43.471122 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:43.471268 8419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:43.472368 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "29b3fa818fad496b9162735934104790"
format_stamp: "Formatted at 2026-09-24 19:30:43 on dist-test-slave-r8cc"
server_key: "199caea46c34d3131ac31a526018ba6e"
server_key_iv: "ea07e602361a716219c43ba27cc8803d"
server_key_version: "encryptionkey@0"
I20260924 19:30:43.472714 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "29b3fa818fad496b9162735934104790"
format_stamp: "Formatted at 2026-09-24 19:30:43 on dist-test-slave-r8cc"
server_key: "199caea46c34d3131ac31a526018ba6e"
server_key_iv: "ea07e602361a716219c43ba27cc8803d"
server_key_version: "encryptionkey@0"
I20260924 19:30:43.473975 8419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:43.474649 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:43.474813 8419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:43.474912 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "29b3fa818fad496b9162735934104790"
format_stamp: "Formatted at 2026-09-24 19:30:43 on dist-test-slave-r8cc"
server_key: "199caea46c34d3131ac31a526018ba6e"
server_key_iv: "ea07e602361a716219c43ba27cc8803d"
server_key_version: "encryptionkey@0"
I20260924 19:30:43.474984 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:43.498293 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:43.500150 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:43.500316 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:43.503938 8419 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:36539
I20260924 19:30:43.503978 8486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:36539 every 8 connection(s)
I20260924 19:30:43.504364 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:43.505193 8487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:43.507480 8487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790: Bootstrap starting.
I20260924 19:30:43.508047 8487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:43.508360 8487 log.cc:824] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:43.509059 8487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790: No bootstrap required, opened a new log
I20260924 19:30:43.509419 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8419
I20260924 19:30:43.509546 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:43.509721 19350 external_mini_cluster.cc:1428] Setting key 33b6848e461ef93930e930784a329044
I20260924 19:30:43.511047 8487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b3fa818fad496b9162735934104790" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36539 } }
I20260924 19:30:43.511206 8487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:43.511312 8487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b3fa818fad496b9162735934104790, State: Initialized, Role: FOLLOWER
I20260924 19:30:43.511474 8487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b3fa818fad496b9162735934104790" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36539 } }
I20260924 19:30:43.511562 8487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:43.511608 8487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:43.511684 8487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:43.512672 8487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b3fa818fad496b9162735934104790" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36539 } }
I20260924 19:30:43.512802 8487 leader_election.cc:302] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b3fa818fad496b9162735934104790; no voters:
I20260924 19:30:43.513041 8487 leader_election.cc:288] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:43.513062 8492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:43.513167 8492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [term 1 LEADER]: Becoming Leader. State: Replica: 29b3fa818fad496b9162735934104790, State: Running, Role: LEADER
I20260924 19:30:43.513316 8487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:43.513304 8492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b3fa818fad496b9162735934104790" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36539 } }
I20260924 19:30:43.513862 8494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b3fa818fad496b9162735934104790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b3fa818fad496b9162735934104790" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36539 } } }
I20260924 19:30:43.513875 8493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b3fa818fad496b9162735934104790. Latest consensus state: current_term: 1 leader_uuid: "29b3fa818fad496b9162735934104790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b3fa818fad496b9162735934104790" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 36539 } } }
I20260924 19:30:43.514038 8494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:43.514050 8493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:43.514278 8501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:43.515261 8501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:30:43 dist-test-slave-r8cc krb5kdc[8406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:43.517009 8501 catalog_manager.cc:1347] Generated new cluster ID: 31be877f77184ad698100973e8ed2ed7
I20260924 19:30:43.517385 8501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:43.521042 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:43.510265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57645 (local address 127.18.229.190:36539)
0924 19:30:43.510422 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:43.510437 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:43.510461 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:30:43.511026 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:43.511032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:43.511074 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:43.511176 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:43.512019 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:43.513098 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:43.514119 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:43.514761 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:43.515206 (+ 445us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:43.517697 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:43.517701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:43.517701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:43.517713 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:43.519382 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:43.519781 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:43.519784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:43.519785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:43.519889 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:43.520140 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:43.520141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:43.520141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:43.520405 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:43.520594 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:43.520675 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:30:43.520771 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260924 19:30:43.522107 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:43.529573 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:43.523550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44487 (local address 127.18.229.190:36539)
0924 19:30:43.523586 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:43.523588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:43.523697 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:30:43.523757 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:43.523758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:43.523768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:43.523816 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:43.524361 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:43.524857 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:43.525555 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:43.525771 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:43.525853 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:43.526965 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:43.526966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:43.526966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:43.526973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:43.528417 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:43.528778 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:43.528779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:43.528779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:43.528837 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:43.529046 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:43.529047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:43.529047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:43.529153 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:43.529258 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:43.529400 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 19:30:43.529461 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:30:43.539543 8501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:43.540259 8501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:43.547842 8501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b3fa818fad496b9162735934104790: Generated new TSK 0
I20260924 19:30:43.548022 8501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:43.553678 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8419
W20260924 19:30:43.557652 8404 connection.cc:535] client connection to 127.18.229.190:36539 recv error: Network error: failed to read from TLS socket (remote: 127.18.229.190:36539): Connection reset by peer (error 104)
2026-09-24T19:30:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:43 dist-test-slave-r8cc krb5kdc[8518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:43 dist-test-slave-r8cc krb5kdc[8518](info): set up 2 sockets
Sep 24 19:30:43 dist-test-slave-r8cc krb5kdc[8518](info): commencing operation
krb5kdc: starting...
W20260924 19:30:44.572337 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:44 dist-test-slave-r8cc krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:44.698603 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34517
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:44.778384 8531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:44.778581 8531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:44.778602 8531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:44.779749 8531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:44.779790 8531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:44.779804 8531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:44.779817 8531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:44.779868 8531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:44.781358 8531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:44.781582 8531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:44.781808 8531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:44.786439 8537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:44.786464 8539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:44.786465 8536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:44.786687 8531 server_base.cc:1034] running on GCE node
I20260924 19:30:44.786954 8531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:44.787235 8531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:44.788393 8531 hybrid_clock.cc:648] HybridClock initialized: now 1790278244788384 us; error 35 us; skew 500 ppm
Sep 24 19:30:44 dist-test-slave-r8cc krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278244, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:44.795145 8531 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:44.795573 8531 webserver.cc:460] Webserver started at http://127.18.229.190:36001/ using document root <none> and password file <none>
I20260924 19:30:44.795812 8531 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:44.795882 8531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:44.796033 8531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:44.797113 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "57da3187e93d4658830c08ac44ab380c"
format_stamp: "Formatted at 2026-09-24 19:30:44 on dist-test-slave-r8cc"
server_key: "d314b7c21be3d18f9f902144eedc5367"
server_key_iv: "dc579d468f276b648c1b07d74a1e8864"
server_key_version: "encryptionkey@0"
I20260924 19:30:44.797474 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "57da3187e93d4658830c08ac44ab380c"
format_stamp: "Formatted at 2026-09-24 19:30:44 on dist-test-slave-r8cc"
server_key: "d314b7c21be3d18f9f902144eedc5367"
server_key_iv: "dc579d468f276b648c1b07d74a1e8864"
server_key_version: "encryptionkey@0"
I20260924 19:30:44.798768 8531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:44.799460 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:44.799624 8531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:44.799723 8531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "57da3187e93d4658830c08ac44ab380c"
format_stamp: "Formatted at 2026-09-24 19:30:44 on dist-test-slave-r8cc"
server_key: "d314b7c21be3d18f9f902144eedc5367"
server_key_iv: "dc579d468f276b648c1b07d74a1e8864"
server_key_version: "encryptionkey@0"
I20260924 19:30:44.799811 8531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:44.816956 8531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:44.819321 8531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:44.819485 8531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:44.823422 8531 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:34517
I20260924 19:30:44.823457 8598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:34517 every 8 connection(s)
I20260924 19:30:44.823809 8531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:44.824688 8599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:44.826870 8599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c: Bootstrap starting.
I20260924 19:30:44.827435 8599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:44.827711 8599 log.cc:824] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:44.828418 8599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c: No bootstrap required, opened a new log
I20260924 19:30:44.829968 8599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57da3187e93d4658830c08ac44ab380c" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34517 } }
I20260924 19:30:44.830103 8599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:44.830147 8599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57da3187e93d4658830c08ac44ab380c, State: Initialized, Role: FOLLOWER
I20260924 19:30:44.830302 8599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57da3187e93d4658830c08ac44ab380c" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34517 } }
I20260924 19:30:44.830365 8599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:44.830386 8599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:44.830452 8599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:44.831267 8599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57da3187e93d4658830c08ac44ab380c" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34517 } }
I20260924 19:30:44.831377 8599 leader_election.cc:302] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57da3187e93d4658830c08ac44ab380c; no voters:
I20260924 19:30:44.831561 8599 leader_election.cc:288] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:44.831620 8602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:44.831770 8602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [term 1 LEADER]: Becoming Leader. State: Replica: 57da3187e93d4658830c08ac44ab380c, State: Running, Role: LEADER
I20260924 19:30:44.831822 8599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:44.831879 8602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57da3187e93d4658830c08ac44ab380c" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34517 } }
I20260924 19:30:44.832304 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57da3187e93d4658830c08ac44ab380c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57da3187e93d4658830c08ac44ab380c" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34517 } } }
I20260924 19:30:44.832309 8604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57da3187e93d4658830c08ac44ab380c. Latest consensus state: current_term: 1 leader_uuid: "57da3187e93d4658830c08ac44ab380c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57da3187e93d4658830c08ac44ab380c" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34517 } } }
I20260924 19:30:44.832412 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:44.832415 8604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:44.832875 8608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:44.833356 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8531
I20260924 19:30:44.833419 8608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:44.833473 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:44.833695 19350 external_mini_cluster.cc:1428] Setting key f93e9de831c9fba5b5ba0b6ec4f6794d
I20260924 19:30:44.835175 8608 catalog_manager.cc:1347] Generated new cluster ID: 9337d18318f14eaca16f73a525573f2f
I20260924 19:30:44.835223 8608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:44.839865 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:44.834164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44559 (local address 127.18.229.190:34517)
0924 19:30:44.834488 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:44.834495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:44.834518 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:30:44.834652 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:44.834660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:44.834743 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:44.834891 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:44.835158 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:44.836559 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:44.837641 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:44.838328 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:44.838437 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:44.839182 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:44.839186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:44.839186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:44.839402 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:44.839441 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:44.839455 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:44.839543 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:44.839625 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:30:44.839655 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":95,"threads_started":1}
I20260924 19:30:44.840811 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:44.841840 8608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:44.842598 8608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:44.845844 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:44.842312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34045 (local address 127.18.229.190:34517)
0924 19:30:44.842350 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:44.842352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:44.842507 (+ 155us) server_negotiation.cc:407] Connection header received
0924 19:30:44.842553 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:44.842554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:44.842592 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:44.842660 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:44.842810 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:44.843534 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:44.844583 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:44.844871 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:44.844931 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:44.845603 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:44.845604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:44.845604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:44.845608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:44.845622 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:44.845625 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:44.845683 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:44.845760 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:30:44.845772 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:30:44.846647 8608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57da3187e93d4658830c08ac44ab380c: Generated new TSK 0
I20260924 19:30:44.846803 8608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:44.849197 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8531
W20260924 19:30:44.853071 8516 connection.cc:535] client connection to 127.18.229.190:34517 recv error: Network error: failed to read from TLS socket (remote: 127.18.229.190:34517): Connection reset by peer (error 104)
2026-09-24T19:30:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:44 dist-test-slave-r8cc krb5kdc[8630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:44 dist-test-slave-r8cc krb5kdc[8630](info): set up 2 sockets
Sep 24 19:30:44 dist-test-slave-r8cc krb5kdc[8630](info): commencing operation
krb5kdc: starting...
W20260924 19:30:45.884085 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:45 dist-test-slave-r8cc krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:45.996786 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42247
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:46.076432 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:46.076654 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:46.076678 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:46.077745 8643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:46.077780 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:46.077793 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:46.077805 8643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:46.077817 8643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:46.079175 8643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:42247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:46.079360 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:46.079520 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:46.084303 8649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:46.084311 8651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:46.084311 8648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:46.084527 8643 server_base.cc:1034] running on GCE node
I20260924 19:30:46.084833 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:46.085100 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:46.086243 8643 hybrid_clock.cc:648] HybridClock initialized: now 1790278246086232 us; error 32 us; skew 500 ppm
Sep 24 19:30:46 dist-test-slave-r8cc krb5kdc[8630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278246, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:46.088762 8643 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:46.089164 8643 webserver.cc:460] Webserver started at http://127.18.229.190:42043/ using document root <none> and password file <none>
I20260924 19:30:46.089382 8643 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:46.089445 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:46.089591 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:46.090622 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "584aa344ab4c4621941b42879ae9dea6"
format_stamp: "Formatted at 2026-09-24 19:30:46 on dist-test-slave-r8cc"
server_key: "b0b00cd0dfa7cba2fb705b3e7127835f"
server_key_iv: "375d859391fbf295c6f995a3a717a9ba"
server_key_version: "encryptionkey@0"
I20260924 19:30:46.091003 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "584aa344ab4c4621941b42879ae9dea6"
format_stamp: "Formatted at 2026-09-24 19:30:46 on dist-test-slave-r8cc"
server_key: "b0b00cd0dfa7cba2fb705b3e7127835f"
server_key_iv: "375d859391fbf295c6f995a3a717a9ba"
server_key_version: "encryptionkey@0"
I20260924 19:30:46.092335 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:46.092995 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:46.093171 8643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:46.093267 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "584aa344ab4c4621941b42879ae9dea6"
format_stamp: "Formatted at 2026-09-24 19:30:46 on dist-test-slave-r8cc"
server_key: "b0b00cd0dfa7cba2fb705b3e7127835f"
server_key_iv: "375d859391fbf295c6f995a3a717a9ba"
server_key_version: "encryptionkey@0"
I20260924 19:30:46.093341 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:46.096603 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:46.098356 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:46.098501 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:46.102278 8643 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:42247
I20260924 19:30:46.102308 8710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:42247 every 8 connection(s)
I20260924 19:30:46.102614 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:46.103579 8711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:46.105865 8711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6: Bootstrap starting.
I20260924 19:30:46.106429 8711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:46.106712 8711 log.cc:824] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:46.107342 8711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6: No bootstrap required, opened a new log
I20260924 19:30:46.109017 8711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584aa344ab4c4621941b42879ae9dea6" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42247 } }
I20260924 19:30:46.109172 8711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:46.109220 8711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584aa344ab4c4621941b42879ae9dea6, State: Initialized, Role: FOLLOWER
I20260924 19:30:46.109381 8711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584aa344ab4c4621941b42879ae9dea6" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42247 } }
I20260924 19:30:46.109443 8711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:46.109483 8711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:46.109531 8711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:46.110364 8711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584aa344ab4c4621941b42879ae9dea6" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42247 } }
I20260924 19:30:46.110554 8711 leader_election.cc:302] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584aa344ab4c4621941b42879ae9dea6; no voters:
I20260924 19:30:46.110769 8711 leader_election.cc:288] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:46.110797 8714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:46.111126 8714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [term 1 LEADER]: Becoming Leader. State: Replica: 584aa344ab4c4621941b42879ae9dea6, State: Running, Role: LEADER
I20260924 19:30:46.111233 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8643
I20260924 19:30:46.111260 8711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:46.111346 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:46.111295 8714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584aa344ab4c4621941b42879ae9dea6" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42247 } }
I20260924 19:30:46.111546 19350 external_mini_cluster.cc:1428] Setting key 9a9a26faf58de188d15a71145b0da975
I20260924 19:30:46.111845 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "584aa344ab4c4621941b42879ae9dea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584aa344ab4c4621941b42879ae9dea6" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42247 } } }
I20260924 19:30:46.112202 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:46.111693 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 584aa344ab4c4621941b42879ae9dea6. Latest consensus state: current_term: 1 leader_uuid: "584aa344ab4c4621941b42879ae9dea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584aa344ab4c4621941b42879ae9dea6" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 42247 } } }
I20260924 19:30:46.112517 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:46.112649 8725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:46.113296 8725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:46.115563 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:46.112132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49931 (local address 127.18.229.190:42247)
0924 19:30:46.113942 (+ 1810us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:46.113951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:46.113986 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:30:46.114097 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:46.114103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:46.114135 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:46.114286 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:46.115068 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:46.115072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:46.115072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:46.115076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:46.115113 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:46.115128 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:46.115200 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:46.115326 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:30:46.115374 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1762,"thread_start_us":76,"threads_started":1}
I20260924 19:30:46.115996 8725 catalog_manager.cc:1347] Generated new cluster ID: c4f0f116a0b3407ba7cc3a29c7c9b407
I20260924 19:30:46.116043 8725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:46.116569 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:46.119262 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:46.118016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54303 (local address 127.18.229.190:42247)
0924 19:30:46.118046 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:46.118047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:46.118159 (+ 112us) server_negotiation.cc:407] Connection header received
0924 19:30:46.118218 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:46.118219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:46.118224 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:46.118280 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:46.118969 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:46.118970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:46.118971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:46.118975 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:46.118990 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:46.118994 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:46.119053 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:46.119183 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:30:46.119204 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:30:46.124703 8725 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:46.124883 8725 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:46.129271 8725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 584aa344ab4c4621941b42879ae9dea6: Generated new TSK 0
I20260924 19:30:46.129468 8725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:46.133737 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8643
2026-09-24T19:30:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:46 dist-test-slave-r8cc krb5kdc[8742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:46 dist-test-slave-r8cc krb5kdc[8742](info): set up 2 sockets
Sep 24 19:30:46 dist-test-slave-r8cc krb5kdc[8742](info): commencing operation
krb5kdc: starting...
W20260924 19:30:47.144218 19350 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:47 dist-test-slave-r8cc krb5kdc[8742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:47.260948 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37793
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:37793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:47.340201 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:47.340428 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:47.340466 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:47.341650 8755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:47.341723 8755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:47.341747 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:47.341770 8755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:47.341799 8755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:47.343284 8755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:37793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:37793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:47.343544 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:47.343781 8755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:47.348490 8763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:47.348542 8755 server_base.cc:1034] running on GCE node
W20260924 19:30:47.348500 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:47.348505 8761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:47.348927 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:47.349212 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:47.350374 8755 hybrid_clock.cc:648] HybridClock initialized: now 1790278247350372 us; error 37 us; skew 500 ppm
Sep 24 19:30:47 dist-test-slave-r8cc krb5kdc[8742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278247, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:47.352945 8755 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:47.353353 8755 webserver.cc:460] Webserver started at http://127.18.229.190:35369/ using document root <none> and password file <none>
I20260924 19:30:47.353579 8755 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:47.353645 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:47.353796 8755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:47.354852 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "6f519ca5957549a7a955bddf7bf3c537"
format_stamp: "Formatted at 2026-09-24 19:30:47 on dist-test-slave-r8cc"
server_key: "ecd1e804d1af7afa53f6f901c97e5cc0"
server_key_iv: "1cee973552f1a30ffa4b52be8cc8a65b"
server_key_version: "encryptionkey@0"
I20260924 19:30:47.355233 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "6f519ca5957549a7a955bddf7bf3c537"
format_stamp: "Formatted at 2026-09-24 19:30:47 on dist-test-slave-r8cc"
server_key: "ecd1e804d1af7afa53f6f901c97e5cc0"
server_key_iv: "1cee973552f1a30ffa4b52be8cc8a65b"
server_key_version: "encryptionkey@0"
I20260924 19:30:47.356585 8755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:47.357283 8770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:47.357439 8755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:47.357534 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "6f519ca5957549a7a955bddf7bf3c537"
format_stamp: "Formatted at 2026-09-24 19:30:47 on dist-test-slave-r8cc"
server_key: "ecd1e804d1af7afa53f6f901c97e5cc0"
server_key_iv: "1cee973552f1a30ffa4b52be8cc8a65b"
server_key_version: "encryptionkey@0"
I20260924 19:30:47.357611 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:47.368573 8755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:47.370741 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:47.370888 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:47.374571 8755 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:37793
I20260924 19:30:47.374624 8822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:37793 every 8 connection(s)
I20260924 19:30:47.374969 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:47.375252 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8755
I20260924 19:30:47.375358 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:47.375540 19350 external_mini_cluster.cc:1428] Setting key c6fbc22efb8550d079dcd32be35476ea
I20260924 19:30:47.375969 8823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:47.379213 8823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537: Bootstrap starting.
I20260924 19:30:47.379846 8823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:47.380167 8823 log.cc:824] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:47.380891 8823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537: No bootstrap required, opened a new log
Sep 24 19:30:47 dist-test-slave-r8cc krb5kdc[8742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:47.383104 8823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f519ca5957549a7a955bddf7bf3c537" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37793 } }
I20260924 19:30:47.383270 8823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:47.383340 8823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f519ca5957549a7a955bddf7bf3c537, State: Initialized, Role: FOLLOWER
I20260924 19:30:47.383468 8823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f519ca5957549a7a955bddf7bf3c537" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37793 } }
I20260924 19:30:47.383525 8823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:47.383601 8823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:47.383654 8823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:47.384337 8823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f519ca5957549a7a955bddf7bf3c537" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37793 } }
I20260924 19:30:47.384464 8823 leader_election.cc:302] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f519ca5957549a7a955bddf7bf3c537; no voters:
I20260924 19:30:47.384650 8823 leader_election.cc:288] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:47.384678 8828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:47.384790 8828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [term 1 LEADER]: Becoming Leader. State: Replica: 6f519ca5957549a7a955bddf7bf3c537, State: Running, Role: LEADER
I20260924 19:30:47.384917 8823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:47.384924 8828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f519ca5957549a7a955bddf7bf3c537" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37793 } }
I20260924 19:30:47.385361 8829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f519ca5957549a7a955bddf7bf3c537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f519ca5957549a7a955bddf7bf3c537" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37793 } } }
I20260924 19:30:47.385385 8830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f519ca5957549a7a955bddf7bf3c537. Latest consensus state: current_term: 1 leader_uuid: "6f519ca5957549a7a955bddf7bf3c537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f519ca5957549a7a955bddf7bf3c537" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 37793 } } }
I20260924 19:30:47.385578 8830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:47.385840 8837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:47.385478 8829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:47.386375 8837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:47.387310 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:47.376049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45539 (local address 127.18.229.190:37793)
0924 19:30:47.376329 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:47.376339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:47.376368 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:30:47.377082 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:47.377091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:47.377168 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:47.377326 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:47.377897 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:47.379258 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:47.380122 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:47.380656 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:47.380781 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:47.383281 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:47.383285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:47.383285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:47.383297 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:47.385291 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:47.385709 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:47.385711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:47.385711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:47.385832 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:47.386058 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:47.386059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:47.386059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:47.386362 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:47.386941 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:47.387009 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:30:47.387147 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260924 19:30:47.388187 8837 catalog_manager.cc:1347] Generated new cluster ID: 699aeaf42b3b47638f19f9c6efdd01df
I20260924 19:30:47.388208 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:47.388237 8837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:47.396548 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:47.389636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42971 (local address 127.18.229.190:37793)
0924 19:30:47.389671 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:47.389673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:47.389817 (+ 144us) server_negotiation.cc:407] Connection header received
0924 19:30:47.389891 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:47.389892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:47.389901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:47.389924 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:47.390542 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:47.391271 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:47.392303 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:47.392613 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:47.392647 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:47.393954 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:47.393955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:47.393955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:47.393961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:47.395470 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:47.395798 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:47.395798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:47.395799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:47.395841 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:47.395998 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:47.395998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:47.395999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:47.396112 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:47.396198 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:47.396390 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:30:47.396467 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:47.405048 8837 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:47.405756 8837 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:47.411551 8837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f519ca5957549a7a955bddf7bf3c537: Generated new TSK 0
I20260924 19:30:47.411739 8837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:47.419788 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8755
2026-09-24T19:30:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:47 dist-test-slave-r8cc krb5kdc[8854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:47 dist-test-slave-r8cc krb5kdc[8854](info): set up 2 sockets
Sep 24 19:30:47 dist-test-slave-r8cc krb5kdc[8854](info): commencing operation
krb5kdc: starting...
W20260924 19:30:48.440296 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:48 dist-test-slave-r8cc krb5kdc[8854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:48.554926 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34901
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:48.634992 8867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:48.635250 8867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:48.635294 8867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:48.636500 8867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:48.636571 8867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:48.636595 8867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:48.636628 8867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:48.636651 8867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:48.638255 8867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:48.638516 8867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:48.638751 8867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:48.643648 8872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:48.643661 8873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:48.643745 8867 server_base.cc:1034] running on GCE node
W20260924 19:30:48.643661 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:48.644155 8867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:48.644505 8867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:48.645643 8867 hybrid_clock.cc:648] HybridClock initialized: now 1790278248645626 us; error 33 us; skew 500 ppm
Sep 24 19:30:48 dist-test-slave-r8cc krb5kdc[8854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:48.648192 8867 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:48.648666 8867 webserver.cc:460] Webserver started at http://127.18.229.190:34709/ using document root <none> and password file <none>
I20260924 19:30:48.648908 8867 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:48.648974 8867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:48.649106 8867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:48.650156 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "2840cfb34a754635b778836a70bddfb7"
format_stamp: "Formatted at 2026-09-24 19:30:48 on dist-test-slave-r8cc"
server_key: "0e9d935c620a06cff5fcdaea1a657f0f"
server_key_iv: "2c635852351d526d1bab502177736616"
server_key_version: "encryptionkey@0"
I20260924 19:30:48.650520 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "2840cfb34a754635b778836a70bddfb7"
format_stamp: "Formatted at 2026-09-24 19:30:48 on dist-test-slave-r8cc"
server_key: "0e9d935c620a06cff5fcdaea1a657f0f"
server_key_iv: "2c635852351d526d1bab502177736616"
server_key_version: "encryptionkey@0"
I20260924 19:30:48.651824 8867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:48.652601 8882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:48.652762 8867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:48.652860 8867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "2840cfb34a754635b778836a70bddfb7"
format_stamp: "Formatted at 2026-09-24 19:30:48 on dist-test-slave-r8cc"
server_key: "0e9d935c620a06cff5fcdaea1a657f0f"
server_key_iv: "2c635852351d526d1bab502177736616"
server_key_version: "encryptionkey@0"
I20260924 19:30:48.652942 8867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:48.660302 8867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:48.662446 8867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:48.662595 8867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:48.666532 8867 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:34901
I20260924 19:30:48.666563 8934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:34901 every 8 connection(s)
I20260924 19:30:48.666908 8867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:48.667692 8935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:48.669790 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8867
I20260924 19:30:48.669919 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:48.670101 19350 external_mini_cluster.cc:1428] Setting key 24b7b97648202ce5dfd6f0c0304f5525
I20260924 19:30:48.670157 8935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7: Bootstrap starting.
I20260924 19:30:48.670935 8935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:48.671254 8935 log.cc:824] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:48.672049 8935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7: No bootstrap required, opened a new log
I20260924 19:30:48.674343 8935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2840cfb34a754635b778836a70bddfb7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34901 } }
I20260924 19:30:48.674503 8935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:48.674535 8935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2840cfb34a754635b778836a70bddfb7, State: Initialized, Role: FOLLOWER
I20260924 19:30:48.674654 8935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2840cfb34a754635b778836a70bddfb7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34901 } }
I20260924 19:30:48.674743 8935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:48.674774 8935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:48.674863 8935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:48.675477 8935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2840cfb34a754635b778836a70bddfb7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34901 } }
I20260924 19:30:48.675585 8935 leader_election.cc:302] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2840cfb34a754635b778836a70bddfb7; no voters:
I20260924 19:30:48.675743 8935 leader_election.cc:288] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:48.675798 8940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:48.675990 8940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [term 1 LEADER]: Becoming Leader. State: Replica: 2840cfb34a754635b778836a70bddfb7, State: Running, Role: LEADER
I20260924 19:30:48.676070 8935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:48.676167 8940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2840cfb34a754635b778836a70bddfb7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34901 } }
I20260924 19:30:48.676309 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:48.670633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54637 (local address 127.18.229.190:34901)
0924 19:30:48.670818 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:48.670827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:48.670860 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:30:48.671448 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:48.671457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:48.671535 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:48.671659 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:48.671892 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:48.673278 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:48.674040 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:48.674632 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:48.674747 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:48.675443 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:48.675447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:48.675448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:48.675651 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:48.675691 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:48.675710 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:48.675792 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:48.675875 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:30:48.675945 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260924 19:30:48.676496 8940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2840cfb34a754635b778836a70bddfb7. Latest consensus state: current_term: 1 leader_uuid: "2840cfb34a754635b778836a70bddfb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2840cfb34a754635b778836a70bddfb7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34901 } } }
I20260924 19:30:48.676505 8942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2840cfb34a754635b778836a70bddfb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2840cfb34a754635b778836a70bddfb7" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34901 } } }
I20260924 19:30:48.676589 8940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:48.676595 8942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:48.677155 8947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:48.677232 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:48.677670 8947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:48.679533 8947 catalog_manager.cc:1347] Generated new cluster ID: 09c8a658613d47c58128fa72f67a9422
I20260924 19:30:48.679597 8947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:48.681890 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:48.678733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38963 (local address 127.18.229.190:34901)
0924 19:30:48.678770 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:48.678773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:48.678907 (+ 134us) server_negotiation.cc:407] Connection header received
0924 19:30:48.679003 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:48.679004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:48.679014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:48.679043 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:48.679263 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:48.679916 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:48.680666 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:48.680897 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:48.680979 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:48.681648 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:48.681649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:48.681650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:48.681654 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:48.681667 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:48.681671 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:48.681718 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:48.681775 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:30:48.681796 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:30:48.709357 8947 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:48.710110 8947 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:48.716822 8947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2840cfb34a754635b778836a70bddfb7: Generated new TSK 0
I20260924 19:30:48.717041 8947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:48.719779 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8867
2026-09-24T19:30:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:48 dist-test-slave-r8cc krb5kdc[8966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:48 dist-test-slave-r8cc krb5kdc[8966](info): set up 2 sockets
Sep 24 19:30:48 dist-test-slave-r8cc krb5kdc[8966](info): commencing operation
krb5kdc: starting...
W20260924 19:30:49.744165 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:49 dist-test-slave-r8cc krb5kdc[8966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:49.858174 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:39829
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:39829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:49.936308 8979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:49.936655 8979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:49.936695 8979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:49.937849 8979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:49.937925 8979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:49.937949 8979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:49.937970 8979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:49.938004 8979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:49.939411 8979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:39829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:49.939600 8979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:49.939817 8979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:49.944500 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:49.944576 8979 server_base.cc:1034] running on GCE node
W20260924 19:30:49.944501 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:49.944533 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:49.944943 8979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:49.945230 8979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:49.946389 8979 hybrid_clock.cc:648] HybridClock initialized: now 1790278249946388 us; error 36 us; skew 500 ppm
Sep 24 19:30:49 dist-test-slave-r8cc krb5kdc[8966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278249, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:49.951894 8979 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:49.952342 8979 webserver.cc:460] Webserver started at http://127.18.229.190:40147/ using document root <none> and password file <none>
I20260924 19:30:49.952533 8979 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:49.952571 8979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:49.952657 8979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:49.953612 8979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "6b5af8841f194ec782581e1ae86fc5a8"
format_stamp: "Formatted at 2026-09-24 19:30:49 on dist-test-slave-r8cc"
server_key: "94171c1488ca43d9fa6f748e6b632d15"
server_key_iv: "1f07f8ece583f1e5bb8b1fa79e8ae9e0"
server_key_version: "encryptionkey@0"
I20260924 19:30:49.953907 8979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "6b5af8841f194ec782581e1ae86fc5a8"
format_stamp: "Formatted at 2026-09-24 19:30:49 on dist-test-slave-r8cc"
server_key: "94171c1488ca43d9fa6f748e6b632d15"
server_key_iv: "1f07f8ece583f1e5bb8b1fa79e8ae9e0"
server_key_version: "encryptionkey@0"
I20260924 19:30:49.955078 8979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:49.955734 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:49.955909 8979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:49.955994 8979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "6b5af8841f194ec782581e1ae86fc5a8"
format_stamp: "Formatted at 2026-09-24 19:30:49 on dist-test-slave-r8cc"
server_key: "94171c1488ca43d9fa6f748e6b632d15"
server_key_iv: "1f07f8ece583f1e5bb8b1fa79e8ae9e0"
server_key_version: "encryptionkey@0"
I20260924 19:30:49.956048 8979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:49.959141 8979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:49.961272 8979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:49.961374 8979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:49.964926 8979 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:39829
I20260924 19:30:49.965014 9046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:39829 every 8 connection(s)
I20260924 19:30:49.965324 8979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:49.966262 9047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:49.968448 9047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8: Bootstrap starting.
I20260924 19:30:49.968986 9047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:49.969280 9047 log.cc:824] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:49.969904 9047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8: No bootstrap required, opened a new log
I20260924 19:30:49.971442 9047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5af8841f194ec782581e1ae86fc5a8" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39829 } }
I20260924 19:30:49.971613 9047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:30:49.971678 9047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5af8841f194ec782581e1ae86fc5a8, State: Initialized, Role: FOLLOWER
I20260924 19:30:49.971859 9047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5af8841f194ec782581e1ae86fc5a8" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39829 } }
I20260924 19:30:49.971959 9047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:30:49.972002 9047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:30:49.972051 9047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:49.972559 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 8979
I20260924 19:30:49.972679 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:49.972875 19350 external_mini_cluster.cc:1428] Setting key be3d363ea2e069f3d0455ea44149073f
I20260924 19:30:49.973083 9047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5af8841f194ec782581e1ae86fc5a8" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39829 } }
I20260924 19:30:49.973198 9047 leader_election.cc:302] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b5af8841f194ec782581e1ae86fc5a8; no voters:
I20260924 19:30:49.973382 9047 leader_election.cc:288] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:30:49.973419 9051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:49.973587 9051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6b5af8841f194ec782581e1ae86fc5a8, State: Running, Role: LEADER
I20260924 19:30:49.973711 9047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:49.973709 9051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5af8841f194ec782581e1ae86fc5a8" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39829 } }
I20260924 19:30:49.974117 9053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b5af8841f194ec782581e1ae86fc5a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5af8841f194ec782581e1ae86fc5a8" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39829 } } }
I20260924 19:30:49.974129 9054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b5af8841f194ec782581e1ae86fc5a8. Latest consensus state: current_term: 1 leader_uuid: "6b5af8841f194ec782581e1ae86fc5a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5af8841f194ec782581e1ae86fc5a8" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 39829 } } }
I20260924 19:30:49.974285 9054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:49.974272 9053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:49.974572 9059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:49.975111 9059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:49.975425 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:49.973293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40317 (local address 127.18.229.190:39829)
0924 19:30:49.973450 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:49.973460 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:49.973589 (+ 129us) server_negotiation.cc:407] Connection header received
0924 19:30:49.973776 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:49.973784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:49.973817 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:49.973932 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:49.974953 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:49.974957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:49.974958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:49.974962 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:49.975004 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:49.975020 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:49.975061 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:49.975202 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:30:49.975257 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260924 19:30:49.976387 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:49.977188 9059 catalog_manager.cc:1347] Generated new cluster ID: b55399ec62c247a3ba809c0d6f36bf73
I20260924 19:30:49.977241 9059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:49.979018 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:49.977926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35761 (local address 127.18.229.190:39829)
0924 19:30:49.977961 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:49.977963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:49.978095 (+ 132us) server_negotiation.cc:407] Connection header received
0924 19:30:49.978158 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:49.978159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:49.978165 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:49.978227 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:49.978789 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:49.978790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:49.978791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:30:49.978795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:49.978808 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:30:49.978812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:49.978862 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:49.978916 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:30:49.978929 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:49.988540 9059 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:49.988685 9059 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:49.995378 9059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b5af8841f194ec782581e1ae86fc5a8: Generated new TSK 0
I20260924 19:30:49.995561 9059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:30:50.001128 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 8979
2026-09-24T19:30:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1281 ms)
[----------] 18 tests from AuthTokenIssuingTest (23339 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:50 dist-test-slave-r8cc krb5kdc[9078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:50 dist-test-slave-r8cc krb5kdc[9078](info): set up 2 sockets
Sep 24 19:30:50 dist-test-slave-r8cc krb5kdc[9078](info): commencing operation
krb5kdc: starting...
W20260924 19:30:51.036445 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:51.153155 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34223
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34223,127.18.229.189:41597,127.18.229.188:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:51.232788 9091 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:30:51.233003 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:51.233021 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:51.233040 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:51.234125 9091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:51.234161 9091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:51.234175 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:51.234187 9091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:51.234200 9091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:51.235587 9091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34223,127.18.229.189:41597,127.18.229.188:34997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:51.235791 9091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:51.235960 9091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:51.240828 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:51.240972 9091 server_base.cc:1034] running on GCE node
W20260924 19:30:51.240828 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:51.241062 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:51.241302 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:51.241564 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:51.242708 9091 hybrid_clock.cc:648] HybridClock initialized: now 1790278251242697 us; error 30 us; skew 500 ppm
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:51.245092 9091 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:51.245577 9091 webserver.cc:460] Webserver started at http://127.18.229.190:42275/ using document root <none> and password file <none>
I20260924 19:30:51.245807 9091 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:51.245875 9091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:51.246026 9091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:51.247097 9091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "1292d8a50d2b4f639798f16e3e3a42c0"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "035681ab1bf300cb3b420325eba24296"
server_key_iv: "a24267b544626c96c7531c70e3dc1f2a"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.247475 9091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "1292d8a50d2b4f639798f16e3e3a42c0"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "035681ab1bf300cb3b420325eba24296"
server_key_iv: "a24267b544626c96c7531c70e3dc1f2a"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.248903 9091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:30:51.249593 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:51.249768 9091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:51.249874 9091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "1292d8a50d2b4f639798f16e3e3a42c0"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "035681ab1bf300cb3b420325eba24296"
server_key_iv: "a24267b544626c96c7531c70e3dc1f2a"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.249948 9091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:51.270859 9091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:51.272815 9091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:51.272986 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:51.277132 9091 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:34223
I20260924 19:30:51.277166 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:34223 every 8 connection(s)
I20260924 19:30:51.277526 9091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:51.277745 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 9091
I20260924 19:30:51.277880 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:51.278116 19350 external_mini_cluster.cc:1428] Setting key 297cab8131d92ae11168290fc18868bc
I20260924 19:30:51.278606 9159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:51.280620 9159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.229.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.189@KRBTEST.COM" created.
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:51.291558 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.281356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34223 (local address 127.0.0.1:59550)
0924 19:30:51.281524 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.281567 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.282069 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.282376 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.282378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.282655 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.282807 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.282812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.283829 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.283830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.284559 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.284562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.285057 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.285058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.285120 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.285656 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.285665 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.288319 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.290090 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.290097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.290099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.290291 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.290537 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.290538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.290538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.290625 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.290968 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.290973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.291034 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.291144 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.291380 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":97,"thread_start_us":76,"threads_started":1}
I20260924 19:30:51.291565 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.281384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.18.229.190:34223)
0924 19:30:51.281563 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.281573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.281678 (+ 105us) server_negotiation.cc:407] Connection header received
0924 19:30:51.282248 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.282250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.282273 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.282371 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.282911 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.283719 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.284657 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.284959 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.285086 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.288413 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.288420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.288420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.288427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.289990 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.290386 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.290389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.290389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.290469 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.290690 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.290691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.290692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.290898 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.290989 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.291174 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.291316 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260924 19:30:51.292270 9159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:51.292626 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.292511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37738)
0924 19:30:51.292557 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.292602 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:30:51.292773 9108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
I20260924 19:30:51.293195 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.293105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37742)
0924 19:30:51.293150 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.293173 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:30:51.293305 9159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:41597: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab.
Entry for principal kudu/127.18.229.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab.
I20260924 19:30:51.300657 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:41597
--webserver_interface=127.18.229.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab
--principal=kudu/127.18.229.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34223,127.18.229.189:41597,127.18.229.188:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:51.342532 9159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } attempt: 1
I20260924 19:30:51.343073 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.342941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37758)
0924 19:30:51.343002 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.343046 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:30:51.343470 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.343385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37762)
0924 19:30:51.343429 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.343454 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:30:51.343577 9159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:41597: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
W20260924 19:30:51.381381 9166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:30:51.381587 9166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:51.381605 9166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:51.381625 9166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:51.382812 9166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:51.382849 9166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:51.382864 9166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:51.382905 9166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:51.382933 9166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:51.384442 9166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34223,127.18.229.189:41597,127.18.229.188:34997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab
--principal=kudu/127.18.229.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.229.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:51.384690 9166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:51.384897 9166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:51.389884 9171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:51.389902 9172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:51.389890 9174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:51.389943 9166 server_base.cc:1034] running on GCE node
I20260924 19:30:51.390295 9166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:51.390573 9166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:51.391747 9166 hybrid_clock.cc:648] HybridClock initialized: now 1790278251391730 us; error 38 us; skew 500 ppm
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:51.394228 9166 init.cc:377] Logged in from keytab as kudu/127.18.229.189@KRBTEST.COM (short username kudu)
I20260924 19:30:51.394640 9166 webserver.cc:460] Webserver started at http://127.18.229.189:44183/ using document root <none> and password file <none>
I20260924 19:30:51.394905 9166 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:51.394951 9166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:51.395094 9166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:51.396200 9166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data/instance:
uuid: "1b05c21f6c314f6189a835e704d4f378"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "695b8b61b941cef097bafab0bd7fd4ab"
server_key_iv: "cabfb2e7f4ca32e0f6ad681bb9acdb0b"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.396559 9166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal/instance:
uuid: "1b05c21f6c314f6189a835e704d4f378"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "695b8b61b941cef097bafab0bd7fd4ab"
server_key_iv: "cabfb2e7f4ca32e0f6ad681bb9acdb0b"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.397836 9166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:30:51.398518 9181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:51.398684 9166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:30:51.398789 9166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal
uuid: "1b05c21f6c314f6189a835e704d4f378"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "695b8b61b941cef097bafab0bd7fd4ab"
server_key_iv: "cabfb2e7f4ca32e0f6ad681bb9acdb0b"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.398861 9166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:51.411711 9159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } attempt: 2
I20260924 19:30:51.412186 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.412044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37774)
0924 19:30:51.412099 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.412153 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:30:51.412551 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.412451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37790)
0924 19:30:51.412501 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.412528 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 19:30:51.412667 9159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:41597: Network error: Client connection negotiation failed: client connection to 127.18.229.189:41597: connect: Connection refused (error 111)
I20260924 19:30:51.420078 9166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:51.422475 9166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:51.422654 9166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:51.426458 9166 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.189:41597
I20260924 19:30:51.426522 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.189:41597 every 8 connection(s)
I20260924 19:30:51.426867 9166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
I20260924 19:30:51.427716 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:51.429229 9234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:51.435482 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 9166
I20260924 19:30:51.435590 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-1/wal/instance
I20260924 19:30:51.435817 19350 external_mini_cluster.cc:1428] Setting key 4371a14b936be4dabd90d09a9755fe81
I20260924 19:30:51.437958 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.429917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.18.229.190:34223)
0924 19:30:51.429994 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.429996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.430112 (+ 116us) server_negotiation.cc:407] Connection header received
0924 19:30:51.430546 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.430547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.430560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.430636 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.431203 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.431686 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.432668 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.432879 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.432942 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.434900 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.434901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.434902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.434908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.436531 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.436917 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.436918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.436919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.436982 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.437273 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.437274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.437275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.437430 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.437485 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.437829 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.437896 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 19:30:51.438230 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.429837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34223 (local address 127.0.0.1:59558)
0924 19:30:51.429989 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.430033 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.430470 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.430709 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.430712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.430945 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.431132 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.431136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.431769 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.431769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.432587 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.432589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.432957 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.432958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.433054 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.433483 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.433492 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.434816 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.436606 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.436612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.436614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.436843 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.437047 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.437047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.437048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.437156 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.437496 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.437502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.437673 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.437808 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.437977 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260924 19:30:51.438825 9234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.229.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.188@KRBTEST.COM" created.
I20260924 19:30:51.446659 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.439102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37804)
0924 19:30:51.439313 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.439324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.439374 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.439503 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.439504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.439662 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.439737 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.439739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.440655 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.440656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.441555 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.441558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.442014 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.442015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.442092 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.442465 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.442470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.443734 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.445553 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.445554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.445555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.445719 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.445932 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.445933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.445933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.445985 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.446322 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.446323 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.446362 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.446432 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.446529 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 19:30:51.446691 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.439170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.18.229.189:41597)
0924 19:30:51.439307 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.439311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.439369 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:30:51.439427 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.439429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.439438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.439491 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.439808 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.440568 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.441637 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.441935 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.441988 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.443832 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.443840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.443841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.443849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.445429 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.445805 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.445809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.445809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.445866 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.446055 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.446056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.446057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.446246 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.446346 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.446462 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.446580 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260924 19:30:51.447244 9234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:51.447593 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.447472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51558)
0924 19:30:51.447521 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.447569 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:30:51.447731 9183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
I20260924 19:30:51.448165 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.448056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51566)
0924 19:30:51.448086 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.448133 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 19:30:51.448309 9234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:34997: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab.
Entry for principal kudu/127.18.229.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab.
I20260924 19:30:51.460088 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:34997
--webserver_interface=127.18.229.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab
--principal=kudu/127.18.229.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:34223,127.18.229.189:41597,127.18.229.188:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:51.497530 9234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } attempt: 1
I20260924 19:30:51.498008 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.497873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51568)
0924 19:30:51.497937 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.497978 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:30:51.498358 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.498261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51576)
0924 19:30:51.498307 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.498333 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:30:51.498440 9234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:34997: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
I20260924 19:30:51.503830 9159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } attempt: 3
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:30:51.511385 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.504309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.18.229.189:41597)
0924 19:30:51.504374 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.504376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.504387 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:30:51.504452 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.504453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.504466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.504536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.504875 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.505382 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.506092 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.506334 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.506421 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.508636 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.508637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.508637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.508644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.510212 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.510676 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.510677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.510677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.510752 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.510956 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.510956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.510956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.511059 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.511109 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.511265 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.511324 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:30:51.511445 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.504236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37816)
0924 19:30:51.504291 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.504304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.504382 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.504543 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.504544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.504716 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.504787 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.504789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.505475 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.505476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.506011 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.506013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.506422 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.506423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.506504 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.506992 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.506996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.508540 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.510297 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.510298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.510299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.510515 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.510827 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.510827 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.510828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.510892 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.511119 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.511120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.511169 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.511251 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.511348 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:30:51.511976 9159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:51.512329 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.512218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51580)
0924 19:30:51.512267 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.512304 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:30:51.512629 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.512550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51592)
0924 19:30:51.512586 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.512610 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:30:51.512729 9159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:34997: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
W20260924 19:30:51.542949 9241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:30:51.543154 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:51.543171 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:51.543191 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:30:51.543877 9159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } attempt: 1
I20260924 19:30:51.544327 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.544193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51600)
0924 19:30:51.544254 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.544297 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:30:51.544713 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.544634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51612)
0924 19:30:51.544672 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.544697 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 19:30:51.544759 9241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:51.544821 9159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:34997: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
W20260924 19:30:51.544869 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:51.544905 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:51.544919 9241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:51.544931 9241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:51.546403 9241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:34223,127.18.229.189:41597,127.18.229.188:34997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab
--principal=kudu/127.18.229.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:34997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.229.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:51.546648 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:51.546893 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:51.551692 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:51.551781 9241 server_base.cc:1034] running on GCE node
W20260924 19:30:51.551682 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:51.551682 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:51.552232 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:51.552525 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:51.553684 9241 hybrid_clock.cc:648] HybridClock initialized: now 1790278251553676 us; error 43 us; skew 500 ppm
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:51.556049 9241 init.cc:377] Logged in from keytab as kudu/127.18.229.188@KRBTEST.COM (short username kudu)
I20260924 19:30:51.556546 9241 webserver.cc:460] Webserver started at http://127.18.229.188:37115/ using document root <none> and password file <none>
I20260924 19:30:51.556797 9241 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:51.556840 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:51.557004 9241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:51.558073 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data/instance:
uuid: "f3572e0642c64ecb98aa5b6366c3707e"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "324a5c797f6791fa499036bf8202bad5"
server_key_iv: "eb933f94460a5a6e4747fe14aad98607"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.558434 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal/instance:
uuid: "f3572e0642c64ecb98aa5b6366c3707e"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "324a5c797f6791fa499036bf8202bad5"
server_key_iv: "eb933f94460a5a6e4747fe14aad98607"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.559718 9241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:30:51.560463 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:51.560632 9241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:51.560727 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal
uuid: "f3572e0642c64ecb98aa5b6366c3707e"
format_stamp: "Formatted at 2026-09-24 19:30:51 on dist-test-slave-r8cc"
server_key: "324a5c797f6791fa499036bf8202bad5"
server_key_iv: "eb933f94460a5a6e4747fe14aad98607"
server_key_version: "encryptionkey@0"
I20260924 19:30:51.560782 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:51.566571 9234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } attempt: 2
I20260924 19:30:51.566989 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.566872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51628)
0924 19:30:51.566927 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.566962 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:30:51.567288 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.567237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51630)
0924 19:30:51.567258 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.567277 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 19:30:51.567380 9234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:34997: Network error: Client connection negotiation failed: client connection to 127.18.229.188:34997: connect: Connection refused (error 111)
I20260924 19:30:51.599928 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:51.602470 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:51.602639 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:51.606564 9241 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.188:34997
I20260924 19:30:51.606614 9308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.188:34997 every 8 connection(s)
I20260924 19:30:51.606998 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
I20260924 19:30:51.607883 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:51.609445 9309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:51.615048 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 9241
I20260924 19:30:51.615131 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790277987605586-19350-0/minicluster-data/master-2/wal/instance
I20260924 19:30:51.615360 19350 external_mini_cluster.cc:1428] Setting key 18607653554dbbd063ba1c95a82890ff
I20260924 19:30:51.618530 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.610170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.18.229.190:34223)
0924 19:30:51.610242 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.610244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.610465 (+ 221us) server_negotiation.cc:407] Connection header received
0924 19:30:51.610905 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.610906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.610914 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.610993 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.611603 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.612068 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.613063 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.613245 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.613295 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.615263 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.615265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.615265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.615273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.617169 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.617614 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.617615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.617615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.617665 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.617918 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.617918 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.617918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.618015 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.618065 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.618352 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.618435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:30:51.618604 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.610147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:34223 (local address 127.0.0.1:59564)
0924 19:30:51.610324 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.610366 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.610829 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.611074 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.611078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.611323 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.611512 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.611516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.612209 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.612210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.612995 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.612997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.613313 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.613314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.613407 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.613836 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.613844 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.615159 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.617244 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.617249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.617252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.617537 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.617733 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.617734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.617734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.617858 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.618076 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.618080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.618224 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.618307 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.618424 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260924 19:30:51.619218 9309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:51.619951 9159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } attempt: 2
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:30:51.626080 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.615802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.18.229.188:34997)
0924 19:30:51.615961 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.615966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.616072 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:30:51.616205 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.616207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.616217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.616284 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.617025 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.618031 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.619045 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.619396 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.619490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.622140 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.622148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.622148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.622155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.623975 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.624417 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.624420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.624421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.624480 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.624692 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.624693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.624693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.624861 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.624991 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.625892 (+ 901us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.625977 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260924 19:30:51.626684 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.619487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:41597 (local address 127.0.0.1:37822)
0924 19:30:51.619587 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.619596 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.619631 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.619752 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.619753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.619898 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.619955 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.619957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.620660 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.620661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.621433 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.621436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.621864 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.621865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.621947 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.622331 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.622336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.623652 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.625405 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.625406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.625407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.625612 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.625845 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.625846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.625846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.625908 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.626401 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.626402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.626447 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.626514 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.626596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:30:51.626701 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.619545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.18.229.189:41597)
0924 19:30:51.619598 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.619600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.619625 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:30:51.619674 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.619675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.619687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.619743 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.620032 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.620569 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.621530 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.621786 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.621859 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.623734 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.623735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.623736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.623743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.625316 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.625691 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.625692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.625692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.625744 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.625976 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.625977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.625977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.626109 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.626174 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.626542 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.626614 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:30:51.627008 9309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:51.630035 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.620200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51646)
0924 19:30:51.620255 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.620264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.620299 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.621688 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.621689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.621813 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.621871 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.621872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.622803 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.622805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.623586 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.623588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.624005 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.624006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.624068 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.624586 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.624590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.626104 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.629016 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.629017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.629017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.629206 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.629392 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.629393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.629393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.629447 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.629717 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.629717 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.629762 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.629853 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.629943 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:30:51.630054 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.620693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.18.229.188:34997)
0924 19:30:51.620879 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.620881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.620890 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:30:51.620909 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.620910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.620921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.620994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.621937 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.622714 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.623672 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.623924 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.624008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.626199 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.626201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.626201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.626209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.628925 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.629283 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.629284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.629284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.629339 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.629545 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.629546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.629546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.629649 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.629729 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.629876 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.629943 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":106,"threads_started":1}
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:30:51.631369 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0: Bootstrap starting.
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:51.632377 9159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:51.632712 9159 log.cc:824] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:51.633971 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0: No bootstrap required, opened a new log
I20260924 19:30:51.635058 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.627244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51648)
0924 19:30:51.627335 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.627346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.627381 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.627537 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.627538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.627703 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.627733 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.627735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.628766 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.628767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.629219 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.629221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.629701 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.629702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.629747 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.630334 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.630338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.632151 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.634025 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.634026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.634027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.634202 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.634403 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.634404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.634404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.634456 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.634738 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.634739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.634775 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.634856 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.634954 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:30:51.635114 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.627332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.18.229.188:34997)
0924 19:30:51.627370 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.627371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.627380 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:30:51.627452 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.627453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.627463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.627505 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.627816 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.628674 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.629308 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.629602 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.629659 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.632255 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.632256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.632257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.632263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.633931 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.634285 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.634286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.634286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.634340 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.634526 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.634527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.634527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.634651 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.634724 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.634889 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.635001 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":20}
I20260924 19:30:51.635645 9159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.635782 9159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1292d8a50d2b4f639798f16e3e3a42c0, State: Initialized, Role: FOLLOWER
I20260924 19:30:51.635869 9159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.636180 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.636245 9159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:51.636394 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: Bootstrap starting.
I20260924 19:30:51.637133 9309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:51.637465 9309 log.cc:824] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:51.637696 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.626943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.18.229.190:34223)
0924 19:30:51.626980 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.626982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.626990 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:51.627031 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.627031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.627041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.627094 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.627653 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.628366 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.629355 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.629652 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.629709 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.632326 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.632327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.632328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.632335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.633600 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.636602 (+ 3002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.636603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.636603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.636663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.636930 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.636931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.636931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.637064 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.637211 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.637499 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.637588 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:30:51.637836 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:51.638418 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: No bootstrap required, opened a new log
W20260924 19:30:51.638509 9328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:51.638572 9328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:51.640635 9309 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.640791 9309 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3572e0642c64ecb98aa5b6366c3707e, State: Initialized, Role: FOLLOWER
I20260924 19:30:51.640864 9309 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.641145 9309 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:51.639087 9091 master_runner.cc:186] Error getting master registration for 127.18.229.189:41597: OK, instance_id { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" instance_seqno: 1790278251425464 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:30:51.642217 9091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:30:51.642791 9330 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.642915 9330 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
W20260924 19:30:51.643601 9341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:51.643698 9341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:51.644094 9241 master_runner.cc:186] Error getting master registration for 127.18.229.189:41597: OK, instance_id { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" instance_seqno: 1790278251425464 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:30:51.646337 9241 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:30:51.647316 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.638328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.18.229.189:41597)
0924 19:30:51.638365 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.638366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.638494 (+ 128us) server_negotiation.cc:407] Connection header received
0924 19:30:51.638518 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.638519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.638530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.638587 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.639215 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.639930 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.640911 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.641210 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.641273 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.644062 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.644063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.644063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.644073 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.645839 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.646213 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.646214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.646214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.646270 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.646459 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.646460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.646460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.646580 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.646746 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.647154 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.647224 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:30:51.648686 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:51.649658 9133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1292d8a50d2b4f639798f16e3e3a42c0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580
I20260924 19:30:51.649806 9133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:30:51.649868 9133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:51.650755 9133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.650959 9133 leader_election.cc:288] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1b05c21f6c314f6189a835e704d4f378 (127.18.229.189:41597), f3572e0642c64ecb98aa5b6366c3707e (127.18.229.188:34997)
I20260924 19:30:51.651311 9283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f3572e0642c64ecb98aa5b6366c3707e"
I20260924 19:30:51.651417 9283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:51.652288 9283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1292d8a50d2b4f639798f16e3e3a42c0 in term 1.
I20260924 19:30:51.652472 9108 leader_election.cc:302] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1292d8a50d2b4f639798f16e3e3a42c0, f3572e0642c64ecb98aa5b6366c3707e; no voters:
I20260924 19:30:51.652588 9316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:51.652755 9316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1292d8a50d2b4f639798f16e3e3a42c0, State: Running, Role: LEADER
I20260924 19:30:51.652849 9316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.653333 9347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1292d8a50d2b4f639798f16e3e3a42c0. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.653585 9347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:51.653920 9349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:51.654115 9349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:51.655824 9283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [term 1 FOLLOWER]: Refusing update from remote peer 1292d8a50d2b4f639798f16e3e3a42c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:51.656167 9347 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:51.657418 9330 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1292d8a50d2b4f639798f16e3e3a42c0. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.657500 9330 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:51.658484 9234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } attempt: 3
Sep 24 19:30:51 dist-test-slave-r8cc krb5kdc[9078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:30:51.665993 9347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.666090 9347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:51.666671 9330 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.666738 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.658727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:34997 (local address 127.0.0.1:51650)
0924 19:30:51.658773 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:51.658784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:51.658820 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:51.658990 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:51.658991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:51.659172 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:51.659232 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.659234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.660174 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.660175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.661004 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:51.661006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.661486 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.661487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:51.661547 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.662038 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:51.662044 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:51.663582 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:51.665628 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.665629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.665629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.665838 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.666104 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:51.666105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:51.666106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:51.666161 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:51.666438 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:51.666439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:51.666483 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:30:51.666546 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:30:51.666639 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:30:51.666761 9330 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:51.667047 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.658788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.18.229.188:34997)
0924 19:30:51.658825 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.658827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.658861 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:30:51.658882 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.658883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.658893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.658938 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.659310 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.660068 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.661083 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.661409 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.661477 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.663682 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.663683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.663684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.663690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.665508 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.665981 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.665982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.665982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.666036 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.666235 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.666236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.666236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.666358 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.666414 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.666891 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.666965 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:51.667475 9350 mvcc.cc:205] Tried to move back new op lower bound from 7332979718779736064 to 7332979718770872320. Current Snapshot: MvccSnapshot[applied={T|T < 7332979718779736064}]
I20260924 19:30:51.668537 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: Bootstrap starting.
I20260924 19:30:51.669261 9234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:51.669562 9234 log.cc:824] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:51.669850 9351 mvcc.cc:205] Tried to move back new op lower bound from 7332979718779736064 to 7332979718770872320. Current Snapshot: MvccSnapshot[applied={T|T < 7332979718779736064}]
I20260924 19:30:51.672685 9349 catalog_manager.cc:1347] Generated new cluster ID: d42c621251f74a23a8b9bc2ae22a8330
I20260924 19:30:51.672739 9349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:51.681746 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.664877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.18.229.190:34223)
0924 19:30:51.664938 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.664940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.668262 (+ 3322us) server_negotiation.cc:407] Connection header received
0924 19:30:51.668371 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.668372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.668383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.668412 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.672792 (+ 4380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.673475 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.674401 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.674672 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.674704 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.678094 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.678095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.678095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.678102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.679977 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.680615 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.680616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.680617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.680676 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.680993 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.680995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.680995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.681137 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.681229 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.681558 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.681640 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:30:51.683177 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.664393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.18.229.189:41597)
0924 19:30:51.664437 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.664439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.665108 (+ 669us) server_negotiation.cc:407] Connection header received
0924 19:30:51.665260 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.665261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.665286 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.665316 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.669035 (+ 3719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.669784 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.671746 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.672054 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.672098 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.674786 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.674787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.674788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.674794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.676390 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.678446 (+ 2056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.678447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.678447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.678501 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.681374 (+ 2873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.681375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.681376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.681493 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.681553 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.683015 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.683091 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:30:51.684580 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:51.664613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.18.229.188:34997)
0924 19:30:51.664648 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.664650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.672328 (+ 7678us) server_negotiation.cc:407] Connection header received
0924 19:30:51.672424 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.672425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.672440 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.672471 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.673367 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.674137 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.675124 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:51.675434 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:51.675518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:51.679076 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:51.679077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:51.679078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:51.679085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:51.681153 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.683776 (+ 2623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.683777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.683777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.683834 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:51.684001 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:51.684001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:51.684002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:51.684126 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:51.684193 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:51.684410 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 19:30:51.684494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:51.685354 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: No bootstrap required, opened a new log
I20260924 19:30:51.687394 9234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.687531 9234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b05c21f6c314f6189a835e704d4f378, State: Initialized, Role: FOLLOWER
I20260924 19:30:51.687610 9234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } }
I20260924 19:30:51.687951 9234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:51.689428 9207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:51.690351 9207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [term 1 FOLLOWER]: Refusing update from remote peer 1292d8a50d2b4f639798f16e3e3a42c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:51.690644 9316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:51.690994 9208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1b05c21f6c314f6189a835e704d4f378"
I20260924 19:30:51.691821 9208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1292d8a50d2b4f639798f16e3e3a42c0 in term 1.
I20260924 19:30:51.694716 9349 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:51.695412 9349 catalog_manager.cc:1504] Loading token signing keys...
W20260924 19:30:51.695847 9370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:51.695955 9370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:51.696096 9374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.696256 9374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:51.696339 9374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1292d8a50d2b4f639798f16e3e3a42c0. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.696431 9374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:51.696496 9374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.696580 9374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:51.697299 9347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1292d8a50d2b4f639798f16e3e3a42c0" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 34223 } } peers { permanent_uuid: "1b05c21f6c314f6189a835e704d4f378" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 41597 } } peers { permanent_uuid: "f3572e0642c64ecb98aa5b6366c3707e" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 34997 } } }
I20260924 19:30:51.697384 9347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:51.698521 9372 mvcc.cc:205] Tried to move back new op lower bound from 7332979718779736064 to 7332979718770872320. Current Snapshot: MvccSnapshot[applied={T|T < 7332979718779736064}]
I20260924 19:30:51.700868 9349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1292d8a50d2b4f639798f16e3e3a42c0: Generated new TSK 0
I20260924 19:30:51.701043 9349 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:30:51.733177 9163 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.731541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.18.229.190:34223)
0924 19:30:51.731600 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.731602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.732512 (+ 910us) server_negotiation.cc:407] Connection header received
0924 19:30:51.732620 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.732622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.732632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.732663 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.733103 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59604: BlockingRecv error: recv got EOF from 127.0.0.1:59604 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 19:30:51.733912 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.731945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.18.229.189:41597)
0924 19:30:51.731984 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.731986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.733421 (+ 1435us) server_negotiation.cc:407] Connection header received
0924 19:30:51.733516 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.733518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.733528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.733557 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.733853 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37846: BlockingRecv error: recv got EOF from 127.0.0.1:37846 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:30:51.734609 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.732271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.18.229.188:34997)
0924 19:30:51.732312 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.732314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.734097 (+ 1783us) server_negotiation.cc:407] Connection header received
0924 19:30:51.734226 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.734228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.734238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.734267 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.734552 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51660: BlockingRecv error: recv got EOF from 127.0.0.1:51660 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:30:51.736604 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.735832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.18.229.189:41597)
0924 19:30:51.735866 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.735868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.736090 (+ 222us) server_negotiation.cc:407] Connection header received
0924 19:30:51.736221 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.736222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.736241 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.736269 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.736558 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37860: BlockingRecv error: recv got EOF from 127.0.0.1:37860 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:30:51.745512 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.744544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.18.229.189:41597)
0924 19:30:51.744599 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.744601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.744924 (+ 323us) server_negotiation.cc:407] Connection header received
0924 19:30:51.745033 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.745034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.745047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.745078 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.745449 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37864: BlockingRecv error: recv got EOF from 127.0.0.1:37864 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 19:30:51.749864 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.749089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.18.229.189:41597)
0924 19:30:51.749125 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.749127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.749353 (+ 226us) server_negotiation.cc:407] Connection header received
0924 19:30:51.749450 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.749451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.749461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.749488 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.749815 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37878: BlockingRecv error: recv got EOF from 127.0.0.1:37878 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:30:51.755988 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.755176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.18.229.189:41597)
0924 19:30:51.755211 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.755213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.755451 (+ 238us) server_negotiation.cc:407] Connection header received
0924 19:30:51.755559 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.755560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.755571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.755600 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.755942 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37892: BlockingRecv error: recv got EOF from 127.0.0.1:37892 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:30:51.766146 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.765301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.18.229.189:41597)
0924 19:30:51.765350 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.765352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.765595 (+ 243us) server_negotiation.cc:407] Connection header received
0924 19:30:51.765694 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.765695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.765706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.765747 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.766086 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37902: BlockingRecv error: recv got EOF from 127.0.0.1:37902 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:30:51.784548 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.783609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.18.229.189:41597)
0924 19:30:51.783653 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.783656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.783921 (+ 265us) server_negotiation.cc:407] Connection header received
0924 19:30:51.784024 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.784026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.784038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.784068 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.784484 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37910: BlockingRecv error: recv got EOF from 127.0.0.1:37910 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:30:51.818763 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.818023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.18.229.189:41597)
0924 19:30:51.818067 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.818070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.818183 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:30:51.818261 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.818262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.818274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.818345 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.818694 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37920: BlockingRecv error: recv got EOF from 127.0.0.1:37920 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:30:51.884784 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:51.884066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.18.229.189:41597)
0924 19:30:51.884173 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:51.884176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:51.884200 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:30:51.884283 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:51.884284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:51.884295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:51.884347 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:51.884730 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37936: BlockingRecv error: recv got EOF from 127.0.0.1:37936 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 19:30:52.014920 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:52.014132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.18.229.189:41597)
0924 19:30:52.014228 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.014230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.014329 (+ 99us) server_negotiation.cc:407] Connection header received
0924 19:30:52.014424 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.014425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.014444 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.014513 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:52.014874 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37940: BlockingRecv error: recv got EOF from 127.0.0.1:37940 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 19:30:52.272903 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:52.272369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.18.229.189:41597)
0924 19:30:52.272435 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.272437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.272447 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:52.272476 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.272477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.272487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.272547 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:52.272855 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37950: BlockingRecv error: recv got EOF from 127.0.0.1:37950 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:30:52.644203 9341 catalog_manager.cc:1259] Loaded cluster ID: d42c621251f74a23a8b9bc2ae22a8330
I20260924 19:30:52.644276 9341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: loading cluster ID for follower catalog manager: success
I20260924 19:30:52.645066 9341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: acquiring CA information for follower catalog manager: success
I20260924 19:30:52.645318 9341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3572e0642c64ecb98aa5b6366c3707e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:30:52.696488 9370 catalog_manager.cc:1259] Loaded cluster ID: d42c621251f74a23a8b9bc2ae22a8330
I20260924 19:30:52.696614 9370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: loading cluster ID for follower catalog manager: success
I20260924 19:30:52.697448 9370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: acquiring CA information for follower catalog manager: success
I20260924 19:30:52.697773 9370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b05c21f6c314f6189a835e704d4f378: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:30:52.789289 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:52.786271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.18.229.189:41597)
0924 19:30:52.786397 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.786399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.786409 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:52.786469 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.786470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.786520 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.786584 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:52.786721 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.787516 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.788578 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.788786 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.788843 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:52.788979 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:52.789085 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:52.789137 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:52.789201 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:30:52.789224 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 19:30:52.793680 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:52.791462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.18.229.188:34997)
0924 19:30:52.791551 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.791553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.791563 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:30:52.791625 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.791625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.791660 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.791729 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:52.791793 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.792335 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.792986 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.793169 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.793227 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:52.793360 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:52.793458 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:52.793513 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:52.793547 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:30:52.793562 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":48,"threads_started":1}
I20260924 19:30:52.798192 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:52.795497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.18.229.188:34997)
0924 19:30:52.795540 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.795542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.795609 (+ 67us) server_negotiation.cc:407] Connection header received
0924 19:30:52.795686 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.795687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.795696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.795742 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:52.795905 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.796629 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.797386 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.797662 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.797724 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:52.797885 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:52.797937 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:52.797988 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:52.798079 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:30:52.798103 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:30:52.798239 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:52.795146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.18.229.189:41597)
0924 19:30:52.795183 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.795185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.795400 (+ 215us) server_negotiation.cc:407] Connection header received
0924 19:30:52.795488 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.795489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.795500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.795564 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:52.795729 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.796483 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.797460 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.797763 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.797828 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:52.797963 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:52.797997 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:52.798046 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:52.798137 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:30:52.798161 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:30:52.798528 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:52.795137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.18.229.190:34223)
0924 19:30:52.795235 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:52.795238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:52.795271 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:30:52.795332 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:52.795333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:52.795386 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:52.795465 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:52.795608 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.796348 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.797439 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:52.797668 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:52.797750 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:52.798036 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:52.798138 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:52.798193 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:52.798415 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 19:30:52.798439 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":49,"threads_started":1}
I20260924 19:30:52.800149 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 9091
I20260924 19:30:52.805554 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 9166
I20260924 19:30:52.810698 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 9241
2026-09-24T19:30:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2811 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:30:52 dist-test-slave-r8cc krb5kdc[9460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:30:52 dist-test-slave-r8cc krb5kdc[9460](info): set up 2 sockets
Sep 24 19:30:52 dist-test-slave-r8cc krb5kdc[9460](info): commencing operation
krb5kdc: starting...
W20260924 19:30:53.845182 19350 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/test-user.keytab.
Sep 24 19:30:53 dist-test-slave-r8cc krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.229.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
Entry for principal kudu/127.18.229.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab.
I20260924 19:30:53.960865 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:35113
--webserver_interface=127.18.229.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:35113,127.18.229.189:34755,127.18.229.188:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:30:54.041644 9473 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:30:54.041924 9473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:54.041956 9473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:54.041998 9473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:54.043193 9473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:54.043262 9473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:54.043287 9473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:54.043308 9473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:54.043340 9473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:54.044914 9473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:35113,127.18.229.189:34755,127.18.229.188:43639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.190.keytab
--principal=kudu/127.18.229.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.190:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.229.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:54.045203 9473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:54.045440 9473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:54.050191 9479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:54.050166 9478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:54.050170 9481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:54.050434 9473 server_base.cc:1034] running on GCE node
I20260924 19:30:54.050649 9473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:54.050938 9473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:54.052101 9473 hybrid_clock.cc:648] HybridClock initialized: now 1790278254052074 us; error 45 us; skew 500 ppm
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:54.054687 9473 init.cc:377] Logged in from keytab as kudu/127.18.229.190@KRBTEST.COM (short username kudu)
I20260924 19:30:54.055109 9473 webserver.cc:460] Webserver started at http://127.18.229.190:33527/ using document root <none> and password file <none>
I20260924 19:30:54.055351 9473 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:54.055420 9473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:54.055564 9473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:54.056684 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data/instance:
uuid: "5db8372b5a714158b3403b8e67e1f3c1"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "0ebef0d3773875273bd96ece0d52329a"
server_key_iv: "373afecb625803e3b78d470c9928bdf1"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.057096 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal/instance:
uuid: "5db8372b5a714158b3403b8e67e1f3c1"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "0ebef0d3773875273bd96ece0d52329a"
server_key_iv: "373afecb625803e3b78d470c9928bdf1"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.058387 9473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:30:54.059072 9488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:54.059239 9473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:54.059345 9473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal
uuid: "5db8372b5a714158b3403b8e67e1f3c1"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "0ebef0d3773875273bd96ece0d52329a"
server_key_iv: "373afecb625803e3b78d470c9928bdf1"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.059419 9473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:54.068432 9473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:54.070680 9473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:54.070855 9473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:54.074774 9473 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.190:35113
I20260924 19:30:54.074810 9540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.190:35113 every 8 connection(s)
I20260924 19:30:54.075333 9473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/data/info.pb
I20260924 19:30:54.076303 9541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:54.077893 9541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
I20260924 19:30:54.085306 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 9473
I20260924 19:30:54.085425 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-0/wal/instance
I20260924 19:30:54.085707 19350 external_mini_cluster.cc:1428] Setting key 2494daf95d125f0d11f344e4277818b0
I20260924 19:30:54.087908 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.078596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:35113 (local address 127.0.0.1:55196)
0924 19:30:54.078763 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.078812 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.079276 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.079611 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.079614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.079871 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.080028 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.080032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.081125 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.081126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.081944 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.081947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.082531 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.082532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.082591 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.083112 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.083120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.084577 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.086433 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.086439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.086441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.086631 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.086885 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.086886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.086887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.086975 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.087325 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.087330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.087391 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.087501 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.087731 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":92,"thread_start_us":71,"threads_started":1}
I20260924 19:30:54.087913 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.078672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.18.229.190:35113)
0924 19:30:54.078855 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.078859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.078906 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:30:54.079452 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.079455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.079472 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.079562 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.080189 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.081026 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.082077 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.082451 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.082544 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.084661 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.084669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.084669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.084676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.086335 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.086736 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.086743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.086743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.086815 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.087042 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.087043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.087044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.087253 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.087346 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.087533 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.087666 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260924 19:30:54.088918 9541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:54.089274 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.089152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58530)
0924 19:30:54.089202 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.089246 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:30:54.089393 9491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
I20260924 19:30:54.089870 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.089751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58546)
0924 19:30:54.089803 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.089835 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 19:30:54.090004 9541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:34755: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.18.229.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab.
Entry for principal kudu/127.18.229.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab.
I20260924 19:30:54.109023 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:34755
--webserver_interface=127.18.229.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab
--principal=kudu/127.18.229.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:35113,127.18.229.189:34755,127.18.229.188:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:54.139252 9541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } attempt: 1
I20260924 19:30:54.139734 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.139596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58548)
0924 19:30:54.139661 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.139700 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:30:54.140086 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.139988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58560)
0924 19:30:54.140036 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.140063 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:30:54.140196 9541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:34755: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
W20260924 19:30:54.190773 9548 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:30:54.190976 9548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:54.190994 9548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:54.191013 9548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:54.192253 9548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:54.192292 9548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:54.192306 9548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:54.192318 9548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:54.192332 9548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:54.193907 9548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:35113,127.18.229.189:34755,127.18.229.188:43639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.189.keytab
--principal=kudu/127.18.229.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.189:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.229.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:54.194164 9548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:54.194393 9548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:54.199287 9556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:54.199360 9554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:54.199360 9553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:54.199410 9548 server_base.cc:1034] running on GCE node
I20260924 19:30:54.199842 9548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:54.200179 9548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:54.201359 9548 hybrid_clock.cc:648] HybridClock initialized: now 1790278254201353 us; error 36 us; skew 500 ppm
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:54.203869 9548 init.cc:377] Logged in from keytab as kudu/127.18.229.189@KRBTEST.COM (short username kudu)
I20260924 19:30:54.204327 9548 webserver.cc:460] Webserver started at http://127.18.229.189:34703/ using document root <none> and password file <none>
I20260924 19:30:54.204519 9548 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:54.204560 9548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:54.204648 9548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:54.205650 9548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data/instance:
uuid: "8970fa7d3280450b8fa33f3dae93bf9f"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "35c25243a7f819170c22c419b025ea0a"
server_key_iv: "7e2af0be689db3bd5dca12c41780da95"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.205947 9548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal/instance:
uuid: "8970fa7d3280450b8fa33f3dae93bf9f"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "35c25243a7f819170c22c419b025ea0a"
server_key_iv: "7e2af0be689db3bd5dca12c41780da95"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.207190 9548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:30:54.207847 9563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:54.208029 9548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:30:54.208114 9548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal
uuid: "8970fa7d3280450b8fa33f3dae93bf9f"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "35c25243a7f819170c22c419b025ea0a"
server_key_iv: "7e2af0be689db3bd5dca12c41780da95"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.208204 9548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:54.208333 9541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } attempt: 2
I20260924 19:30:54.208734 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.208593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58564)
0924 19:30:54.208669 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.208707 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:30:54.209066 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.208969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58576)
0924 19:30:54.209017 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.209042 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:30:54.209167 9541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.189:34755: Network error: Client connection negotiation failed: client connection to 127.18.229.189:34755: connect: Connection refused (error 111)
I20260924 19:30:54.234175 9548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:54.236434 9548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:54.236613 9548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:54.240489 9548 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.189:34755
I20260924 19:30:54.240520 9615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.189:34755 every 8 connection(s)
I20260924 19:30:54.240901 9548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/data/info.pb
I20260924 19:30:54.241801 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:54.243330 9616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:54.243827 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 9548
I20260924 19:30:54.243927 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-1/wal/instance
I20260924 19:30:54.244166 19350 external_mini_cluster.cc:1428] Setting key 1fe878698dd2333d2608ee339a0fc020
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.229.188@KRBTEST.COM; defaulting to no policy
I20260924 19:30:54.253527 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.244223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.18.229.190:35113)
0924 19:30:54.244283 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.244286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.244560 (+ 274us) server_negotiation.cc:407] Connection header received
0924 19:30:54.245311 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.245312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.245322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.245365 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.246092 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.246866 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.247937 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.248323 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.248391 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.250650 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.250651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.250652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.250660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.252097 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.252617 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.252618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.252618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.252669 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.252892 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.252892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.252892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.253003 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.253088 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.253399 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.253464 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:30:54.253705 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.244233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:35113 (local address 127.0.0.1:55200)
0924 19:30:54.244408 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.244460 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.245193 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.245435 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.245439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.245802 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.245999 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.246004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.246968 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.246968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.247837 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.247839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.248458 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.248459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.248604 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.249228 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.249237 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.250552 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.252239 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.252246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.252248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.252545 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.252731 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.252731 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.252732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.252834 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.253133 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.253142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.253269 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.253370 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.253493 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260924 19:30:54.254317 9616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.229.188@KRBTEST.COM" created.
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:30:54.262384 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.254630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58578)
0924 19:30:54.254717 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.254729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.254778 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.255117 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.255118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.255261 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.255308 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.255309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.256274 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.256276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.256767 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.256769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.257433 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.257434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.257494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.257890 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.257894 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.259372 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.261090 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.261091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.261091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.261317 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.261536 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.261537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.261537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.261637 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.261913 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.261914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.261964 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.262078 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.262184 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:30:54.262486 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.254789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.18.229.189:34755)
0924 19:30:54.254956 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.254960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.254980 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:30:54.255003 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.255005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.255015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.255115 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.255417 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.256186 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.256864 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.257348 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.257423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.259456 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.259463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.259464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.259469 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.260999 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.261404 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.261406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.261407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.261458 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.261708 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.261709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.261709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.261841 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.261903 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.262106 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.262291 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260924 19:30:54.263171 9616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:54.263585 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.263448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38142)
0924 19:30:54.263475 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.263556 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 19:30:54.263741 9567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
I20260924 19:30:54.264230 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.264098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38154)
0924 19:30:54.264173 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.264213 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:30:54.264354 9616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:43639: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.229.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab.
Entry for principal kudu/127.18.229.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab.
I20260924 19:30:54.269220 19350 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2KIbLR/build/release/bin/kudu
/tmp/dist-test-task2KIbLR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:43639
--webserver_interface=127.18.229.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab
--principal=kudu/127.18.229.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.229.148:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.229.190:35113,127.18.229.189:34755,127.18.229.188:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:30:54.300302 9541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } attempt: 3
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:30:54.308017 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.300663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58584)
0924 19:30:54.300780 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.300794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.300860 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.301023 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.301024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.301224 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.301299 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.301301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.301957 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.301959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.302810 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.302813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.303183 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.303184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.303260 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.303717 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.303723 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.305134 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.307098 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.307099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.307099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.307265 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.307465 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.307466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.307466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.307517 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.307780 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.307780 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.307819 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.307872 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.307945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:30:54.308185 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.300785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.18.229.189:34755)
0924 19:30:54.300844 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.300846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.300858 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:30:54.300927 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.300927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.300939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.300991 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.301381 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.301871 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.302900 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.303109 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.303163 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.305230 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.305232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.305232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.305242 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.307005 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.307336 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.307337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.307338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.307400 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.307587 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.307588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.307588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.307717 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.307782 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.307974 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.308065 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:30:54.308836 9541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:54.309195 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.309093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38160)
0924 19:30:54.309141 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.309172 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:30:54.309617 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.309512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38174)
0924 19:30:54.309554 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.309597 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:30:54.309727 9541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:43639: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
I20260924 19:30:54.313587 9616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } attempt: 1
I20260924 19:30:54.313956 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.313854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38180)
0924 19:30:54.313906 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.313936 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 19:30:54.314292 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.314215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38194)
0924 19:30:54.314253 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.314277 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:30:54.314392 9616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:43639: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
I20260924 19:30:54.340916 9541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } attempt: 1
I20260924 19:30:54.341378 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.341246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38208)
0924 19:30:54.341311 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.341349 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:30:54.341750 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.341649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38216)
0924 19:30:54.341696 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.341733 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:30:54.341861 9541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:43639: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
W20260924 19:30:54.353061 9623 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:30:54.353294 9623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:30:54.353334 9623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:30:54.353368 9623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:30:54.354547 9623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:30:54.354611 9623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:30:54.354638 9623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:30:54.354671 9623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:30:54.354698 9623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:30:54.356209 9623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.229.148:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.229.190:35113,127.18.229.189:34755,127.18.229.188:43639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/krb5kdc/kudu_127.18.229.188.keytab
--principal=kudu/127.18.229.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.229.188:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.229.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:30:54.356477 9623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:30:54.356693 9623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:30:54.361366 9629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:54.361416 9631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:30:54.361392 9628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:30:54.361806 9623 server_base.cc:1034] running on GCE node
I20260924 19:30:54.362094 9623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:30:54.362383 9623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:30:54.363559 9623 hybrid_clock.cc:648] HybridClock initialized: now 1790278254363561 us; error 44 us; skew 500 ppm
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:30:54.366034 9623 init.cc:377] Logged in from keytab as kudu/127.18.229.188@KRBTEST.COM (short username kudu)
I20260924 19:30:54.366465 9623 webserver.cc:460] Webserver started at http://127.18.229.188:42233/ using document root <none> and password file <none>
I20260924 19:30:54.366700 9623 fs_manager.cc:359] Metadata directory not provided
I20260924 19:30:54.366771 9623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:30:54.366918 9623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:30:54.367985 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data/instance:
uuid: "d4d925d705d244ba920df819b72b7c9b"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "80b9f5712a1014a8882fc29571798d0a"
server_key_iv: "f317f4ad3591270b2ad0cf0ad12b1cb6"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.368409 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal/instance:
uuid: "d4d925d705d244ba920df819b72b7c9b"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "80b9f5712a1014a8882fc29571798d0a"
server_key_iv: "f317f4ad3591270b2ad0cf0ad12b1cb6"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.369733 9623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:30:54.370429 9638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:54.370590 9623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:30:54.370693 9623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data,/tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal
uuid: "d4d925d705d244ba920df819b72b7c9b"
format_stamp: "Formatted at 2026-09-24 19:30:54 on dist-test-slave-r8cc"
server_key: "80b9f5712a1014a8882fc29571798d0a"
server_key_iv: "f317f4ad3591270b2ad0cf0ad12b1cb6"
server_key_version: "encryptionkey@0"
I20260924 19:30:54.370769 9623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:30:54.382561 9616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } attempt: 2
I20260924 19:30:54.383101 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.382942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38226)
0924 19:30:54.383015 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.383063 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 19:30:54.383472 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.383387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38240)
0924 19:30:54.383431 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.383456 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:30:54.383589 9616 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.229.188:43639: Network error: Client connection negotiation failed: client connection to 127.18.229.188:43639: connect: Connection refused (error 111)
I20260924 19:30:54.395200 9623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:30:54.397353 9623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:30:54.397527 9623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:30:54.401468 9623 rpc_server.cc:307] RPC server started. Bound to: 127.18.229.188:43639
I20260924 19:30:54.401501 9690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.229.188:43639 every 8 connection(s)
I20260924 19:30:54.401875 9623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/data/info.pb
I20260924 19:30:54.402765 9691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:30:54.403657 19350 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2KIbLR/build/release/bin/kudu as pid 9623
I20260924 19:30:54.403764 19350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2KIbLR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790277987605586-19350-0/minicluster-data/master-2/wal/instance
I20260924 19:30:54.403973 19350 external_mini_cluster.cc:1428] Setting key aa93df5b003a3e82a205e8bf5b53a720
I20260924 19:30:54.404841 9691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:30:54.414358 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.405926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.18.229.190:35113)
0924 19:30:54.405986 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.405988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.406483 (+ 495us) server_negotiation.cc:407] Connection header received
0924 19:30:54.406511 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.406512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.406522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.406589 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.407026 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.407689 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.408603 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.408811 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.408898 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.411253 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.411254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.411255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.411262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.412969 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.413422 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.413423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.413424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.413479 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.413742 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.413743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.413744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.413857 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.413930 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.414188 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.414267 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:30:54.414561 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.405867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.190:35113 (local address 127.0.0.1:55204)
0924 19:30:54.406394 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.406428 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.406474 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.406615 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.406617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.406848 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.406943 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.406948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.407778 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.407779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.408503 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.408506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.408907 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.408908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.408984 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.409556 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.409564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.411163 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.413086 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.413093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.413095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.413338 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.413541 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.413542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.413542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.413651 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.413936 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.413942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.414065 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.414165 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.414318 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":83,"threads_started":1}
I20260924 19:30:54.414907 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.404507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.18.229.188:43639)
0924 19:30:54.404665 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.404674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.404844 (+ 170us) server_negotiation.cc:407] Connection header received
0924 19:30:54.405436 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.405446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.405510 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.405669 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.406372 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.407525 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.408365 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.408685 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.408811 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.411771 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.411779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.411779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.411786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.413257 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.413602 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.413605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.413605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.413681 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.413883 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.413884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.413885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.414068 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.414212 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.414299 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.414376 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260924 19:30:54.415128 9691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:54.417001 9541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } attempt: 2
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.190@KRBTEST.COM
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.190@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:30:54.422659 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.415421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.189:34755 (local address 127.0.0.1:58598)
0924 19:30:54.415460 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.415468 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.415504 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.415668 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.415669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.415783 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.415817 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.415818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.416514 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.416515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.417028 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.417030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.417692 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.417692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.417767 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.418133 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.418137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.419274 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.421224 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.421224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.421225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.421430 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.421666 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.421666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.421666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.421722 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.422316 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.422317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.422358 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.422417 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.422524 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:30:54.422782 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.415474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.18.229.189:34755)
0924 19:30:54.415517 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.415519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.415526 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:30:54.415552 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.415553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.415563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.415620 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.415898 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.416427 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.417288 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.417573 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.417614 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.419358 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.419359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.419359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.419367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.421143 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.421531 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.421532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.421533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.421591 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.421800 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.421802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.421802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.422234 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.422542 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.422588 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.422682 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:30:54.423393 9691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:30:54.424533 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.415635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.18.229.190:35113)
0924 19:30:54.415670 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.415672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.415679 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:30:54.415718 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.415719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.415726 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.415772 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.416261 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.416694 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.417542 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.417744 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.417826 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.421193 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.421194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.421195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.421201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.423034 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.423376 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.423378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.423378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.423433 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.423824 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.423826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.423826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.423947 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.424098 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.424183 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.424445 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:30:54.425148 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.417238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38254)
0924 19:30:54.417290 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.417301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.417341 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.417592 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.417593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.417761 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.417801 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.417803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.418669 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.418670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.419485 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.419487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.419970 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.419972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.420036 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.420605 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.420611 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.422045 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.424131 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.424131 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.424132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.424363 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.424526 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.424527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.424527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.424598 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.424878 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.424879 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.424921 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.424983 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.425085 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:30:54.425482 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.417412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.18.229.188:43639)
0924 19:30:54.417468 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.417470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.417477 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:30:54.417494 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.417495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.417503 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.417873 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.417901 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.418566 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.419572 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.419895 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.419952 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.422130 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.422131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.422131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.422138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.424039 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.424426 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.424427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.424427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.424481 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.424672 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.424673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.424673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.424794 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.424873 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.425340 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.425402 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:30:54.427737 9541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1: Bootstrap starting.
I20260924 19:30:54.428551 9541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:54.428889 9541 log.cc:824] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.188@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.229.189@KRBTEST.COM
I20260924 19:30:54.429738 9541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1: No bootstrap required, opened a new log
I20260924 19:30:54.432199 9541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.432377 9541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db8372b5a714158b3403b8e67e1f3c1, State: Initialized, Role: FOLLOWER
I20260924 19:30:54.432487 9541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.432790 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.432847 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.423660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38268)
0924 19:30:54.423834 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.423843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.423875 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.424539 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.424540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.424652 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.424682 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.424683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.426275 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.426276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.426966 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.426970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.427436 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.427437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.427489 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.427882 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.427887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.429889 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.431790 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.431791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.431792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.431978 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.432209 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.432210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.432210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.432266 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.432560 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.432561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.432599 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.432661 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.432754 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260924 19:30:54.433128 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.423734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.18.229.188:43639)
0924 19:30:54.424311 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.424313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.424319 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:30:54.424338 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.424339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.424349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.424424 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.424927 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.426217 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.427060 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.427360 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.427430 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.429973 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.429973 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.429974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.429979 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.431703 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.432064 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.432065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.432065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.432143 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.432329 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.432329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.432329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.432492 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.432555 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.432953 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.433025 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":103,"threads_started":1}
I20260924 19:30:54.432852 9541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:54.433380 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.425220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.18.229.189:34755)
0924 19:30:54.425262 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.425264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.425271 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:30:54.425486 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.425487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.425497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.425556 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.426082 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.426598 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.427251 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.427517 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.427587 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.430003 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.430004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.430005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.430012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.431738 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.432175 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.432177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.432177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.432239 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.432458 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.432459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.432460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.432590 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.432758 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.433207 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.433280 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:30:54.433113 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.434633 19350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:30:54.434898 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: Bootstrap starting.
I20260924 19:30:54.435541 9691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:54.435427 9515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5db8372b5a714158b3403b8e67e1f3c1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55208
I20260924 19:30:54.435812 9515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:30:54.435871 9515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:54.435868 9691 log.cc:824] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: Log is configured to *not* fsync() on all Append() calls
W20260924 19:30:54.435992 9709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:54.436053 9709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:54.436786 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: No bootstrap required, opened a new log
I20260924 19:30:54.436800 9515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.437250 9515 leader_election.cc:288] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [CANDIDATE]: Term 1 election: Requested vote from peers 8970fa7d3280450b8fa33f3dae93bf9f (127.18.229.189:34755), d4d925d705d244ba920df819b72b7c9b (127.18.229.188:43639)
I20260924 19:30:54.438714 9691 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.438854 9691 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d925d705d244ba920df819b72b7c9b, State: Initialized, Role: FOLLOWER
I20260924 19:30:54.438941 9691 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.439242 9691 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:54.439314 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5db8372b5a714158b3403b8e67e1f3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d4d925d705d244ba920df819b72b7c9b"
I20260924 19:30:54.439410 9665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:54.440264 9665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5db8372b5a714158b3403b8e67e1f3c1 in term 1.
I20260924 19:30:54.439239 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.440548 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.438745 9473 master_runner.cc:186] Error getting master registration for 127.18.229.189:34755: OK, instance_id { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" instance_seqno: 1790278254239365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:30:54.441185 9473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:30:54.441403 9492 leader_election.cc:302] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db8372b5a714158b3403b8e67e1f3c1, d4d925d705d244ba920df819b72b7c9b; no voters:
I20260924 19:30:54.441558 9698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:30:54.441735 9698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5db8372b5a714158b3403b8e67e1f3c1, State: Running, Role: LEADER
I20260924 19:30:54.441816 9698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.442212 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db8372b5a714158b3403b8e67e1f3c1. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.442291 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: This master's current role is: LEADER
W20260924 19:30:54.441157 9727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:54.443328 9727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:54.443511 9731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:30:54.443799 9731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:30:54.444231 9623 master_runner.cc:186] Error getting master registration for 127.18.229.189:34755: OK, instance_id { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" instance_seqno: 1790278254239365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:30:54.446486 9623 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:30:54.456650 9665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [term 1 FOLLOWER]: Refusing update from remote peer 5db8372b5a714158b3403b8e67e1f3c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:54.457059 9729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:54.458663 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db8372b5a714158b3403b8e67e1f3c1. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.458761 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.460197 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.460287 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:54.460343 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.460417 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.460714 9735 mvcc.cc:205] Tried to move back new op lower bound from 7332979730250133504 to 7332979730194280448. Current Snapshot: MvccSnapshot[applied={T|T < 7332979730250133504}]
I20260924 19:30:54.465314 9731 catalog_manager.cc:1347] Generated new cluster ID: 042b82791f0145169d1c4d735bf0d0ac
I20260924 19:30:54.465376 9731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:30:54.470353 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.442460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.18.229.190:35113)
0924 19:30:54.442492 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.442494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.448475 (+ 5981us) server_negotiation.cc:407] Connection header received
0924 19:30:54.448537 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.448538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.448550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.448581 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.452194 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.452929 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.453918 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.454221 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.454291 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.463370 (+ 9079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.463372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.463372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.463380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.468482 (+ 5102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.469199 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.469201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.469201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.469263 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.469583 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.469584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.469584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.469747 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.469843 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.470171 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.470257 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":17}
I20260924 19:30:54.472908 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.442400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.18.229.189:34755)
0924 19:30:54.442447 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.442449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.448331 (+ 5882us) server_negotiation.cc:407] Connection header received
0924 19:30:54.450131 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.450133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.450146 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.450205 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.451678 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.452432 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.454806 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.455132 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.455199 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.462468 (+ 7269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.462470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.462470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.462478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.464334 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.471695 (+ 7361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.471696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.471697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.471783 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.472168 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.472170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.472170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.472316 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.472397 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.472729 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.472815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:30:54.473584 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.442157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.18.229.188:43639)
0924 19:30:54.442193 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.442195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.448716 (+ 6521us) server_negotiation.cc:407] Connection header received
0924 19:30:54.448747 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.448748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.448761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.448823 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.450152 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.450933 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.455936 (+ 5003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.456313 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.456378 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.463078 (+ 6700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.463079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.463079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.463086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.465182 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.472021 (+ 6839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.472022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.472023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.472098 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.472564 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.472565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.472565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.472747 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.472827 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.473406 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.473495 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:54.474743 9616 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } attempt: 3
I20260924 19:30:54.476356 9736 mvcc.cc:205] Tried to move back new op lower bound from 7332979730250133504 to 7332979730194280448. Current Snapshot: MvccSnapshot[applied={T|T < 7332979730250133504}]
Sep 24 19:30:54 dist-test-slave-r8cc krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278254, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.229.189@KRBTEST.COM for kudu/127.18.229.188@KRBTEST.COM
I20260924 19:30:54.483184 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.475004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.229.188:43639 (local address 127.0.0.1:38282)
0924 19:30:54.475111 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 19:30:54.475123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:30:54.475169 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:30:54.475338 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:30:54.475339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:30:54.475510 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:30:54.475556 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.475558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.476470 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.476471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.477305 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:30:54.477308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.477789 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.477790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:30:54.477847 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.478289 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:30:54.478294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:30:54.479756 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:30:54.481946 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.481948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.481948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.482165 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.482400 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:30:54.482401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:30:54.482401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:30:54.482461 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:30:54.482744 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:30:54.482745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:30:54.482791 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:30:54.482986 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 19:30:54.483084 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:30:54.483187 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:54.475095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.18.229.188:43639)
0924 19:30:54.475130 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.475131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.475156 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:30:54.475226 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.475227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.475237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.475293 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.475642 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.476395 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.477388 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:54.477714 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:54.477803 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:54.479852 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:30:54.479853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:30:54.479853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:30:54.479860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:30:54.481871 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.482244 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.482245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.482246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.482301 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:30:54.482520 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:30:54.482521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:30:54.482521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:30:54.482654 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:30:54.482743 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:54.483023 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 19:30:54.483096 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:30:54.485301 9616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: Bootstrap starting.
I20260924 19:30:54.486060 9616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: Neither blocks nor log segments found. Creating new log.
I20260924 19:30:54.486358 9616 log.cc:824] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:30:54.501111 9616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: No bootstrap required, opened a new log
I20260924 19:30:54.503134 9616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.503264 9616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8970fa7d3280450b8fa33f3dae93bf9f, State: Initialized, Role: FOLLOWER
I20260924 19:30:54.503340 9616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } }
I20260924 19:30:54.503674 9616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:30:54.504388 9589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:30:54.505275 9589 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [term 1 FOLLOWER]: Refusing update from remote peer 5db8372b5a714158b3403b8e67e1f3c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:30:54.505571 9698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:30:54.506526 9590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5db8372b5a714158b3403b8e67e1f3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8970fa7d3280450b8fa33f3dae93bf9f"
I20260924 19:30:54.507181 9747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.507443 9747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.507505 9747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db8372b5a714158b3403b8e67e1f3c1. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.507556 9747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.507588 9747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.507645 9747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:30:54.508147 9731 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:30:54.508358 9590 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5db8372b5a714158b3403b8e67e1f3c1 for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 19:30:54.508869 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5db8372b5a714158b3403b8e67e1f3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5db8372b5a714158b3403b8e67e1f3c1" member_type: VOTER last_known_addr { host: "127.18.229.190" port: 35113 } } peers { permanent_uuid: "8970fa7d3280450b8fa33f3dae93bf9f" member_type: VOTER last_known_addr { host: "127.18.229.189" port: 34755 } } peers { permanent_uuid: "d4d925d705d244ba920df819b72b7c9b" member_type: VOTER last_known_addr { host: "127.18.229.188" port: 43639 } } }
I20260924 19:30:54.508970 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:30:54.509104 9731 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:30:54.512804 9745 mvcc.cc:205] Tried to move back new op lower bound from 7332979730454507520 to 7332979730194280448. Current Snapshot: MvccSnapshot[applied={T|T < 7332979730250133504}]
W20260924 19:30:54.516716 9757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:30:54.516768 9757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:30:54.528162 9731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5db8372b5a714158b3403b8e67e1f3c1: Generated new TSK 0
I20260924 19:30:54.528394 9731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:30:54.550967 9696 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.549273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.18.229.188:43639)
0924 19:30:54.549319 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.549321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.550294 (+ 973us) server_negotiation.cc:407] Connection header received
0924 19:30:54.550400 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.550401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.550413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.550445 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.550901 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38296: BlockingRecv error: recv got EOF from 127.0.0.1:38296 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:30:54.551903 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.549699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.18.229.189:34755)
0924 19:30:54.549739 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.549741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.551319 (+ 1578us) server_negotiation.cc:407] Connection header received
0924 19:30:54.551416 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.551417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.551428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.551457 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.551842 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58612: BlockingRecv error: recv got EOF from 127.0.0.1:58612 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:30:54.552855 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.550028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.18.229.190:35113)
0924 19:30:54.550066 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.550068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.552216 (+ 2148us) server_negotiation.cc:407] Connection header received
0924 19:30:54.552385 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.552387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.552398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.552428 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.552797 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55232: BlockingRecv error: recv got EOF from 127.0.0.1:55232 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:30:54.555274 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.554522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.18.229.189:34755)
0924 19:30:54.554562 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.554563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.554783 (+ 220us) server_negotiation.cc:407] Connection header received
0924 19:30:54.554884 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.554885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.554901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.554930 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.555233 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58622: BlockingRecv error: recv got EOF from 127.0.0.1:58622 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:30:54.558252 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.557498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.18.229.189:34755)
0924 19:30:54.557531 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.557533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.557787 (+ 254us) server_negotiation.cc:407] Connection header received
0924 19:30:54.557875 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.557876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.557886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.557913 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.558211 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58636: BlockingRecv error: recv got EOF from 127.0.0.1:58636 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:30:54.562090 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.561322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.18.229.189:34755)
0924 19:30:54.561354 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.561355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.561581 (+ 226us) server_negotiation.cc:407] Connection header received
0924 19:30:54.561693 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.561694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.561704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.561732 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.562052 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58642: BlockingRecv error: recv got EOF from 127.0.0.1:58642 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:30:54.568257 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.567430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.18.229.189:34755)
0924 19:30:54.567462 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.567464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.567703 (+ 239us) server_negotiation.cc:407] Connection header received
0924 19:30:54.567827 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.567828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.567838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.567867 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.568203 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58654: BlockingRecv error: recv got EOF from 127.0.0.1:58654 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:30:54.578280 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.577445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.18.229.189:34755)
0924 19:30:54.577492 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.577494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.577745 (+ 251us) server_negotiation.cc:407] Connection header received
0924 19:30:54.577848 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.577849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.577863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.577892 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.578230 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58664: BlockingRecv error: recv got EOF from 127.0.0.1:58664 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:30:54.596657 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.595670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.18.229.189:34755)
0924 19:30:54.595716 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.595719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.596007 (+ 288us) server_negotiation.cc:407] Connection header received
0924 19:30:54.596146 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.596147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.596161 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.596192 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.596571 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58670: BlockingRecv error: recv got EOF from 127.0.0.1:58670 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:30:54.630790 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.630020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.18.229.189:34755)
0924 19:30:54.630070 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.630073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.630226 (+ 153us) server_negotiation.cc:407] Connection header received
0924 19:30:54.630315 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.630316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.630328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.630371 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.630724 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58686: BlockingRecv error: recv got EOF from 127.0.0.1:58686 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:30:54.696879 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.696087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.18.229.189:34755)
0924 19:30:54.696161 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.696164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.696279 (+ 115us) server_negotiation.cc:407] Connection header received
0924 19:30:54.696359 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.696360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.696373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.696449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.696812 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58698: BlockingRecv error: recv got EOF from 127.0.0.1:58698 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 19:30:54.826767 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:54.826145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.18.229.189:34755)
0924 19:30:54.826207 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:54.826209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:54.826234 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:30:54.826298 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:54.826299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:54.826309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:54.826371 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:54.826705 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58704: BlockingRecv error: recv got EOF from 127.0.0.1:58704 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 19:30:55.084882 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:30:55.084110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.18.229.189:34755)
0924 19:30:55.084208 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:55.084210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:55.084224 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:30:55.084299 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:55.084300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:55.084310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:55.084393 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:30:55.084822 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58714: BlockingRecv error: recv got EOF from 127.0.0.1:58714 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:30:55.443826 9727 catalog_manager.cc:1259] Loaded cluster ID: 042b82791f0145169d1c4d735bf0d0ac
I20260924 19:30:55.443902 9727 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: loading cluster ID for follower catalog manager: success
I20260924 19:30:55.444700 9727 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: acquiring CA information for follower catalog manager: success
I20260924 19:30:55.444972 9727 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4d925d705d244ba920df819b72b7c9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:30:55.517227 9757 catalog_manager.cc:1259] Loaded cluster ID: 042b82791f0145169d1c4d735bf0d0ac
I20260924 19:30:55.517308 9757 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: loading cluster ID for follower catalog manager: success
I20260924 19:30:55.518091 9757 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: acquiring CA information for follower catalog manager: success
I20260924 19:30:55.518329 9757 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8970fa7d3280450b8fa33f3dae93bf9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:30:55.600759 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:55.597954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.18.229.189:34755)
0924 19:30:55.598090 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:55.598093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:55.598167 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:30:55.598281 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:55.598282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:55.598349 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:55.598436 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:55.598590 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.599101 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.599873 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.600079 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.600199 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:55.600394 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:55.600509 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:55.600601 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:55.600626 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:30:55.600647 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260924 19:30:55.604806 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:55.602426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.18.229.188:43639)
0924 19:30:55.602542 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:55.602544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:55.602552 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:30:55.602589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:55.602590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:55.602626 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:55.602699 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:55.602783 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.603337 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.604063 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.604318 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.604371 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:55.604494 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:55.604594 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:55.604648 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:55.604661 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 19:30:55.604676 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260924 19:30:55.609720 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:55.606805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.18.229.189:34755)
0924 19:30:55.606949 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:55.606951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:55.607064 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:30:55.607169 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:55.607170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:55.607181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:55.607209 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:55.607413 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.608047 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.608989 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.609274 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.609331 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:55.609482 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:55.609513 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:55.609558 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:55.609612 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:30:55.609630 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260924 19:30:55.609879 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:55.606862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.18.229.188:43639)
0924 19:30:55.606906 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:55.606908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:55.607057 (+ 149us) server_negotiation.cc:407] Connection header received
0924 19:30:55.607093 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:55.607094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:55.607104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:55.607159 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:55.607281 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.607954 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.609048 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.609338 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.609404 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:55.609536 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:55.609565 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:55.609607 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:55.609767 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:30:55.609787 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:30:55.610437 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:30:55.606854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.18.229.190:35113)
0924 19:30:55.607099 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:30:55.607102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:30:55.607230 (+ 128us) server_negotiation.cc:407] Connection header received
0924 19:30:55.607276 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:30:55.607277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:30:55.607325 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:30:55.607425 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:30:55.607604 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.608406 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.609342 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:30:55.609661 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:30:55.609717 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:30:55.609932 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:30:55.610063 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:30:55.610124 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:30:55.610347 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 19:30:55.610371 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":55,"threads_started":1}
I20260924 19:30:55.616430 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 9473
I20260924 19:30:55.621771 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 9548
I20260924 19:30:55.626555 19350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2KIbLR/build/release/bin/kudu with pid 9623
2026-09-24T19:30:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2815 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5626 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268026 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST